Merge pull request #16 from th3r00t/development

First Alpha Release 0.1.0

.gitignore

@@ -1,14 +1,16 @@
-app/books/*
-*.json
+books/*
+*.epub
+*.idea
 *.pyc
 app/content.opf
 .vscode
-app/data/catalogue.db
+app/data/*
+fontend/db.sqlite3
 GPATH
 GRTAGS
 GTAGS
 .#*
-
+frontend/interface/migrations/*
 # Standard Python gitignore below
 # Created by https://www.gitignore.io/api/python
 # Edit at https://www.gitignore.io/?templates=python

.pre-commit-config.yaml

@@ -22,3 +22,10 @@ repos:
     hooks:
       - id: isort
         additional_dependencies: ["toml"]
+
+  # Python code formatting
+  - repo: https://github.com/psf/black
+    rev: stable
+    hooks:
+      - id: black
+        language_version: python3.8

README.md

@@ -1,5 +1,8 @@
 # pyShelf 0.1.0
-A simple terminal based ebook server
+
+<p align="center"><b>A simple terminal based ebook server</b></p>
+<a href="https://asciinema.org/a/M739CljirFAf9nzeNyNO0113a" target="_blank"><img src="https://asciinema.org/a/M739CljirFAf9nzeNyNO0113a.svg" /></a>
+<img src="https://raw.githubusercontent.com/th3r00t/pyShelf/development/src/interface/static/img/pyShelf_frontend_0_1_0.png" alt="Server Frontend" align="center" />
 
 Frustrated with Calibre being my only option for hosting my eBook collection, I have decided to spin up my own.
 
@@ -17,23 +20,38 @@ pyShelf uses [`pre-commit`](https://pre-commit.com/) to automate some tasks.
 Before developing, run `pre-commit install`.
 See the [documentation](https://pre-commit.com/) for more information.
 
-pyShelf uses ['Doxygen'](https://http://www.doxygen.nl/) for source code documentation.
+pyShelf uses ['Doxygen'](http://www.doxygen.nl/) for source code documentation.
 Any changes to source should be documented and have run doxygen doxygen.conf prior to commiting.
 
 pyShelf follows ['sem-ver'](https://semver.org) standards. Before advancing version numbers be sure to set PROJECT_NUMBER in doxygen.conf accordingly.
 
 ## Configuration
-All configuration is done in config.py.
-The only currently required configuration is to set book_path to the location of your books.
+All pyShelf configuration is done in config.py.
+
+### Nginx configuration
+I have included a default nginx config file pyshelf_nginx.conf. This file should be sufficient to get you up and running. You are required to change the location alias's to reflect your pyshelf install folder leaving everything after /frontend intact.
+
+Further resources for nginx setup may be found @ [This nginx, django, & uwsgi, guide](https://uwsgi-docs.readthedocs.io/en/latest/tutorials/Django_and_nginx.html)
+
+### uwsgi configuration
+Inside uwsgi.ini you should make changes to reflect your install directory, and the port you wish uwsgi to listen on. Alternativly you can make the requisite changes to listen on a socket instead. This change would also require a change to the pyshelf_nginx.conf file as well.
+
+### pyShelf configuration
+User configuration is contained within config.json in the project root. The only currently required configuration is to set book_path to the location of your books.
 
 ## Current Features
-Currently pyShelf will recursively scan your collection, extract and store some metadata in the sqlite database.
+Currently pyShelf will recursively scan your collection, extract and store some metadata in the sqlite database. It will also provide you with a web based frontend to view and download your books. Note that this is a very early alpha and lacking the ability to sort and search your collection. This feature is coming however.
 
-Django is being implemented to power the frontend experience, and web based database maintenance. The first steps of which are included in this commit. Also the book database has been switched over to reflect this.
+Django has been implemented to power the frontend experience, and web based database maintenance. The first steps of which are included in this commit. Also the book database has been switched over to reflect this. A properly configured web server is required for hosting the frontend, configuration of which is outside of the scope of this readme. Running via the Django test server might be possible, albeit not recomended.
+
+## In Progress
+
+* UI/UX tweaks, including making the book display responsive. and not so ugly.
+* Searching, & further organizational tools.
+* Improved cover image storage, and acquisition.
 
 ## Future Goals
-* HTML Frontend for file transfers
-* HTML Backend for catalogue maintenance
+* Support for other book formats (Currently on supporting EPUBS)
 * Terminal Backend for catalogue maintenance
 * Calculate page count from total characters
   * (Thanks to @Fireblend for the idea) https://github.com/th3r00t/pyShelf/issues/3

__init__.py

@@ -0,0 +1,4 @@
+import sys
+
+sys.path.insert(1, "app/")
+sys.path.insert(2, "frontend/")

app/__init__.py

@@ -1,4 +0,0 @@
-import os
-import sys
-
-sys.path.insert(0, os.path.abspath('.'))

app/config.py

@@ -1,13 +0,0 @@
-class Config:
-    """Main System Configuration"""
-    def __init__(self):
-        self.book_path = "books/"
-        self.TITLE = "pyShelf E-Book Server"
-        self.book_shelf = "data/shelf.json"
-        # self.catalogue_db = "data/catalogue.db"
-        self.catalogue_db = "../frontend/db.sqlite3"
-        self.file_array = [
-           self.book_shelf,
-           self.catalogue_db,
-           ]
-        self.auto_scan = True

app/lib/display.py

@@ -1,103 +0,0 @@
-#!/usr/bin/python
-import cgi
-import sys
-
-from config import Config
-
-sys.path.insert(0, '../')
-
-
-class Frontend():
-    """Dynamic frontend display functions"""
-
-    def __init__(self, dimensions=[0, 0]):
-        """
-        :param dimensions: array containing screen size [x, y]
-        """
-        self.dimensions = dimensions
-        self.TITLE = Config().TITLE
-
-    def html_Headers(self):
-        """
-        HTML headers
-        :returns _head: HTML render of page headers
-        """
-        _head = """
-        <!DOCTYPE html>
-        <html lang=\"en\">
-        <head>
-        <meta charset=\"utf-8\">
-        <meta http-equiv=\"X-UA-Compatible\" content=\"IE-edge\">
-        <meta name=\"viewport\" content=\"width=device-width, initial-scale=1\">
-        <link type=\"text/css\" rel=\"stylesheet\" href=\"/css/main.css\" />
-        <title>%s</title>
-        </head>
-        """ % self.TITLE
-        return _head
-
-    def app_Headers(self):
-        """
-        App specific headers
-        :returns _head: HTML render of application specific headers
-        """
-        _head = """
-        <body>
-        <div id=\"app\">
-        <div class=\"app_header\">
-        <h1 class=\"app_hdr shadow\">pyShelf</h1>
-        <h2> class=\"app_subhdr shadow\">Open Source E-book Server</h2>
-        </div>
-        """
-        return _head
-
-    def app_body(self, nav, shelf):
-        """
-        Main interface body, and navigation
-        :param nav: nav[] system navigation list
-        :param shelf: shelf[0{path:"",title:"",cover:"",author:""}]
-        :returns _body: HTML render of page body
-        """
-        _body = """
-        <div class=\"app_body\">
-        <div class=\"left_col\">
-        %s
-        </div>
-        <div class=\"shelf\">
-        <div class=\"shelf_contents\">
-        %s
-        </div>
-        </div>
-        </div>
-        """ %(nav, shelf)
-        return _body
-
-    def app_footer(self):
-        """
-        Main interface footer; Closes HTML
-        :returns _footer: HTML render of page footer
-        """
-        _footer = """
-        <div class=\"app_footer\">
-        <div class=\"python_logo\">
-        <img src=\"/img/py.png\" id=\"python_logo\" />
-        </div>
-        </div>
-        </div>
-        </body>
-        </html>
-        """
-        return _footer
-
-    def compile(self, nav, shelf):
-        """
-        Compiles user interface
-        :returns _ui: Compiled HTML for page layout
-        """
-        _head = self.html_Headers() + self.app_Headers()
-        _body = self.app_body(nav, shelf)
-        _foot = self.app_footer()
-        try:
-            _ui = _head + _body + _foot
-            return _ui
-        except Exception as e:
-            return e

app/lib/library.py

@@ -1,151 +0,0 @@
-#!/usr/bin/python
-import json
-import os
-import re
-import zipfile
-
-from bs4 import BeautifulSoup
-from PIL import Image
-
-from config import Config
-from lib.api_hooks import DuckDuckGo
-from lib.storage import Storage
-
-config = Config()
-
-
-class Catalogue:
-    """Decodes and stores book information"""
-    """Step One: filter_books"""
-
-    def __init__(self):
-        self.file_list = []
-        self.opf_regx = re.compile(r'\.opf')
-        self.cover_regx = re.compile(r'\.jpg|\.jpeg|\.png|\.bmp|\.gif')
-        self.html_regx = re.compile(r'\.html')
-
-    def scan_folder(self, folder=config.book_path):
-        for f in os.listdir(folder):
-            _path = os.path.abspath(folder+'/'+f)
-            #_path = os.path.abspath('.')+'/'+folder+f+'/'
-            _is_dir = os.path.isdir(_path.strip()+'/')
-            if _is_dir:
-                self.file_list.append(self.scan_folder(_path))
-            self.file_list.append(_path)
-
-    def scan_book(self, book):
-        """REMOVE ME?"""
-        _epub = zipfile.ZipFile(book)
-        with _epub as _epub_open:
-            try: _epub_open.open('content.opf'); return True
-            except Exception as e: print(e); return False
-
-    def filter_books(self):
-        """
-            Scan book folder recursively for epub files
-            filter_books(0) -> Catalogue.books
-            filter_books(1) -> self.books[]
-            :param ret: 0 -> create class property -> dump json
-            :param ret: 1 -> create & return class property
-        """
-        self.scan_folder()
-        regx = re.compile(r"\.epub")
-        self.books = list(filter(regx.search, filter(None, self.file_list)))
-        _book_list_expanded = {}
-        with open(config.book_shelf, 'w') as f:
-            for book in self.books:
-                _book_list_expanded[book] = self.process_book(book)
-            json.dump(_book_list_expanded, f)
-        return _book_list_expanded
-
-    def process_book(self, book):
-        """Return dictionary of epub file contents"""
-        f_name = 'content.opf'
-        book = zipfile.ZipFile(book, 'r')
-        details = {}
-        with book as book_zip:
-            details['files'] = []
-            details['path'] = book.filename
-            expanded = book_zip.infolist()
-            regx = re.compile(r'\.opf|cover')
-            for i in expanded:
-                match = re.search(regx, i.filename)
-                if match:
-                    # Returns zip file location of requested files
-                    details['files'].append(match.string)
-        return details
-
-    def extract_metadata(self, book):
-        """
-        Return extracted metadata and cover picture
-        book['path'] == Full path to ebook file
-        book['files'] == list of files from self.process_book(book)
-        """
-        book_zip = zipfile.ZipFile(book['path'], 'r')
-        with book_zip as f:
-            content = self.extract_content(book_zip, book)
-            soup = BeautifulSoup(content, "lxml")
-            title = soup.find("dc:title")
-            if title == None:
-                title = book['path'].split('/')[-1].rsplit('.', 1)[0]
-            else: title = title.contents[0]
-            author = soup.find("dc:creator")
-            if author != None: author = author.contents[0]
-            try: cover = self.extract_cover_image(book_zip, book)
-            except IndexError:
-                # cover = self.extract_cover_html(book_zip, book)
-                cover = DuckDuckGo().image_result(title)
-            book_details = [title, author, cover, book['path']]
-        return book_details
-
-    def extract_content(self, book_zip, book):
-        content = book_zip.open(
-            list(
-                filter(self.opf_regx.search, book['files'])
-            )[0]
-        )
-        return content
-
-    def extract_cover_html(self, book_zip, book):
-        cover = book_zip.open(
-            list(
-                filter(self.html_regx.search, book['files'])
-            )[0]
-        )
-        return cover
-
-    def extract_cover_image(self, book_zip, book):
-        cover = book_zip.open(
-            list(
-                filter(self.cover_regx.search, book['files'])
-            )[0]
-        )
-        try: cover = book_zip.read(cover.name); return cover
-        except KeyError: return False
-
-    def compare_shelf_current(self):
-        db = Storage()
-        stored = db.book_paths_list()
-        closed = db.close()
-        try: self.books
-        except Exception: self.filter_books()
-        on_disk, in_storage = [], []
-        for _x in self.books: on_disk.append(_x)
-        for _y in stored: in_storage.append(_y[0])
-        a, b, = set(on_disk), set(in_storage)
-        c = set.difference(a, b)
-        return c
-
-    def import_books(self, list=None):
-        book_list = self.compare_shelf_current()
-        db = Storage()
-        for book in book_list:
-            book = self.process_book(book)
-            extracted = self.extract_metadata(book)
-            db.insert_book(extracted)
-        inserted = db.commit()
-        if inserted is not True:
-            print(inserted)
-            if input('Continue ? y/n') == 'y':
-                pass
-        db.close()

app/lib/pyShelf.py

@@ -1,149 +0,0 @@
-#!/usr/bin/python
-import mimetypes
-import os
-import zipfile
-from http.server import BaseHTTPRequestHandler, HTTPServer
-
-from config import Config
-from lib.library import Catalogue
-from lib.storage import Storage
-
-config = Config()
-Storage = Storage()
-
-
-class InitFiles:
-    """First run file creation operations"""
-    def __init__(self, file_array):
-        print("Begining creation of file structure")
-        for _pointer in file_array:
-            if not os.path.isfile(_pointer):
-                self.CreateFile(_pointer)
-        print("Concluded file creation")
-
-    def CreateFile(self, _pointer):
-        """Create the file"""
-        if not os.path.isdir(os.path.split(_pointer)[0]):
-            os.mkdir(os.path.split(_pointer)[0])
-            f = open(_pointer, "w+")
-            f.close()
-
-
-class RequestHandler(BaseHTTPRequestHandler):
-    """Request Handler"""
-    def do_GET(self):
-        # TODO determine how to include stylesheets
-        self.send_response(200)
-        if self.path == '/':
-            self.path = '../static/index.html'
-            mimetype = 'text/html'
-            serve_file = open(self.path[1:]).read()
-            self.send_header('Content-type', mimetype)
-            self.end_headers()
-            self.wfile.write(bytes(serve_file, 'utf-8'))
-        elif self.path.split('.', 1)[1] == 'css':
-            self.path = '../static' + self.path
-            mimetype = 'text/css'
-            serve_file = open(self.path[1:]).read()
-            self.send_header('Content-type', mimetype)
-            self.end_headers()
-            self.wfile.write(bytes(serve_file, 'utf-8'))
-        elif self.path.endswith('.png'):
-            self.path = '../static' + self.path
-            mimetype = 'image/png'
-            serve_file = open(self.path[1:], 'rb')
-            # Important to rb read binary for images
-            self.send_header('Content-type', mimetype)
-            self.end_headers()
-            self.wfile.write(serve_file.read())
-        else:
-            self.send_response(404)
-            serve_file = "File Not Found"
-            mimetype = 'text/html'
-            self.send_header('Content-type', mimetype)
-            self.end_headers()
-        try: serve_file.close()
-        except Exception: pass
-
-
-class BookDisplay:
-    """All functions related to displaying book information in the HTML UI"""
-
-    def __init__(self):
-        """
-        Initialize class variables
-        :return: None
-        """
-        self.books_per_page = None
-        self.current_page = 0
-        self.thumbnail_size = [200, 300]
-        self.thumbnail_scale = 1
-        self.total_pages = None
-
-    def nextPage(self):
-        """
-        Goto next book page
-        :return: new current_page
-        """
-        self.current_page += 1
-        return self.current_page
-
-    def previousPage(self):
-        """
-        Goto previous book page
-        :return: new current_page
-        """
-        self.current_page -= 1
-        return self.current_page
-
-    def booksPerPage(self, screen_size):
-        """
-        Set books per page
-
-        :param screen_size: Array containing x,y pixel sizes
-        :return: self.books_per_page
-        """
-        x = (self.thumbnail_size[0] * self.thumbnail_scale) + 10
-        y = (self.thumbnail_size[1] * self.thumbnail_scale) + 10
-        self.books_per_page = int(screen_size[0]//x) * int(screen_size[1]//y)
-        return self.books_per_page
-
-
-class BookServer:
-    """
-    HTTP server functions required to display e-books
-    """
-
-    def __init__(self):
-        self.server_address = ('', 8000)
-        self.handler = RequestHandler
-        self.httpd = HTTPServer(self.server_address, self.handler)
-
-    def close_prompt(self):
-        """Prompt to close server"""
-        close = input("Close Server? y/n")
-        if close == 'y':
-            self.close()
-            return True
-        else:
-            self.close_prompt()
-
-    def run(self):
-        """Start HTTP Server"""
-        try:
-            print("Server running @ http://127.0.0.1:8000")
-            self.httpd.serve_forever()
-            self.httpd.handle_request()
-        except KeyboardInterrupt:
-            print("Interrupt received, Closing Server")
-            self.close()
-            print("Server shutdown, Goodbye!")
-            return True
-
-    def close(self):
-        """Stop HTTP Server"""
-        try:
-            self.httpd.server_close()
-            return True
-        except Exception:
-            return False

app/main.py

@@ -1,21 +0,0 @@
-#!/usr/bin/python
-import sys
-
-from config import Config
-from lib.display import Frontend
-from lib.library import Catalogue
-from lib.pyShelf import BookDisplay, BookServer, InitFiles
-
-# sys.path.insert(1, 'lib/')
-
-config = Config()  # Get configuration settings
-InitFiles(config.file_array)  # Initialize file system
-Catalogue = Catalogue()  # Open the Catalogue
-UI = Frontend()
-Server = BookServer()
-# new_books = Catalogue.new_files()
-Catalogue.import_books()  # Filter Your books
-# Server.run()
-# TODO Figure out a system to get books page count
-# TODO Update Documentation
-# TODO Requirements.txt

config.json

@@ -0,0 +1,7 @@
+{
+  "TITLE": "pyShelf E-Book Server",
+  "VERSION": "0.1.0",
+  "BOOKPATH": "books/",
+  "DATABASE": "src/db.sqlite3",
+  "BOOKSHELF": "data/shelf.json"
+}

frontend/frontend/settings.py

@@ -1,121 +0,0 @@
-"""
-Django settings for frontend project.
-
-Generated by 'django-admin startproject' using Django 2.2.7.
-
-For more information on this file, see
-https://docs.djangoproject.com/en/2.2/topics/settings/
-
-For the full list of settings and their values, see
-https://docs.djangoproject.com/en/2.2/ref/settings/
-"""
-
-import os
-
-# Build paths inside the project like this: os.path.join(BASE_DIR, ...)
-BASE_DIR = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
-
-
-# Quick-start development settings - unsuitable for production
-# See https://docs.djangoproject.com/en/2.2/howto/deployment/checklist/
-
-# SECURITY WARNING: keep the secret key used in production secret!
-SECRET_KEY = '@(9b9jslgg41u1u=mr)-2*-n2x0vef0zsy39*z@sz18&tvow18'
-
-# SECURITY WARNING: don't run with debug turned on in production!
-DEBUG = True
-
-ALLOWED_HOSTS = []
-
-
-# Application definition
-
-INSTALLED_APPS = [
-    'django.contrib.admin',
-    'django.contrib.auth',
-    'django.contrib.contenttypes',
-    'django.contrib.sessions',
-    'django.contrib.messages',
-    'django.contrib.staticfiles',
-    'interface',
-]
-
-MIDDLEWARE = [
-    'django.middleware.security.SecurityMiddleware',
-    'django.contrib.sessions.middleware.SessionMiddleware',
-    'django.middleware.common.CommonMiddleware',
-    'django.middleware.csrf.CsrfViewMiddleware',
-    'django.contrib.auth.middleware.AuthenticationMiddleware',
-    'django.contrib.messages.middleware.MessageMiddleware',
-    'django.middleware.clickjacking.XFrameOptionsMiddleware',
-]
-
-ROOT_URLCONF = 'frontend.urls'
-
-TEMPLATES = [
-    {
-        'BACKEND': 'django.template.backends.django.DjangoTemplates',
-        'DIRS': [],
-        'APP_DIRS': True,
-        'OPTIONS': {
-            'context_processors': [
-                'django.template.context_processors.debug',
-                'django.template.context_processors.request',
-                'django.contrib.auth.context_processors.auth',
-                'django.contrib.messages.context_processors.messages',
-            ],
-        },
-    },
-]
-
-WSGI_APPLICATION = 'frontend.wsgi.application'
-
-
-# Database
-# https://docs.djangoproject.com/en/2.2/ref/settings/#databases
-
-DATABASES = {
-    'default': {
-        'ENGINE': 'django.db.backends.sqlite3',
-        'NAME': os.path.join(BASE_DIR, 'db.sqlite3'),
-    }
-}
-
-
-# Password validation
-# https://docs.djangoproject.com/en/2.2/ref/settings/#auth-password-validators
-
-AUTH_PASSWORD_VALIDATORS = [
-    {
-        'NAME': 'django.contrib.auth.password_validation.UserAttributeSimilarityValidator',
-    },
-    {
-        'NAME': 'django.contrib.auth.password_validation.MinimumLengthValidator',
-    },
-    {
-        'NAME': 'django.contrib.auth.password_validation.CommonPasswordValidator',
-    },
-    {
-        'NAME': 'django.contrib.auth.password_validation.NumericPasswordValidator',
-    },
-]
-
-
-# Internationalization
-# https://docs.djangoproject.com/en/2.2/topics/i18n/
-
-LANGUAGE_CODE = 'en-us'
-
-TIME_ZONE = 'UTC'
-
-USE_I18N = True
-
-USE_L10N = True
-
-USE_TZ = True
-
-
-# Static files (CSS, JavaScript, Images)
-# https://docs.djangoproject.com/en/2.2/howto/static-files/
-
-STATIC_URL = '/static/'

frontend/interface/migrations/0001_initial.py

@@ -1,27 +0,0 @@
-# Generated by Django 2.2.7 on 2019-11-10 03:56
-
-from django.db import migrations, models
-
-
-class Migration(migrations.Migration):
-
-    initial = True
-
-    dependencies = [
-    ]
-
-    operations = [
-        migrations.CreateModel(
-            name='books',
-            fields=[
-                ('id', models.AutoField(auto_created=True, primary_key=True, serialize=False, verbose_name='ID')),
-                ('title', models.CharField(max_length=255)),
-                ('author', models.CharField(blank=True, max_length=255)),
-                ('categories', models.CharField(blank=True, max_length=255)),
-                ('cover', models.BinaryField(blank=True, editable=True)),
-                ('pages', models.IntegerField(blank=True)),
-                ('progress', models.IntegerField(blank=True)),
-                ('file_name', models.CharField(max_length=255)),
-            ],
-        ),
-    ]

frontend/interface/models.py

@@ -1,23 +0,0 @@
-from django.db import models
-
-# Create your models here.
-
-
-class books(models.Model):
-    """
-    pyShelfs Book Database class
-    :param title: Book title
-    :param author: Author
-    :param categories: Categories <-- Not implemented
-    :param cover: Cover image BinaryField
-    :param pages: # of pages <-- Not implemented
-    :param progress: Reader percentage <-- Not implented
-    :param file_name: Path to book
-    """
-    title = models.CharField(max_length=255)
-    author = models.CharField(max_length=255, blank=True)
-    categories = models.CharField(max_length=255, blank=True)
-    cover = models.BinaryField(blank=True, editable=True)
-    pages = models.IntegerField(blank=True)
-    progress = models.IntegerField(blank=True)
-    file_name = models.CharField(max_length=255, blank=False)

frontend/interface/views.py

@@ -1,6 +0,0 @@
-from django.shortcuts import render
-from django.http import HttpResponse
-# Create your views here.
-
-def index(request):
-    return HttpResponse("HTTP Request")

importBooks

@@ -0,0 +1,12 @@
+#!/usr/bin/python
+
+import pathlib
+import sys
+
+from src.backend.pyShelf_ScanLibrary import execute_scan
+
+PRG_PATH = pathlib.Path.cwd()
+LIB_PATH = pathlib.Path.joinpath(PRG_PATH, "src", "backend", "lib")
+sys.path.insert(0, PRG_PATH)
+print("\n")
+execute_scan(PRG_PATH)

pyproject.toml

@@ -7,4 +7,4 @@ use_parentheses = true
 # NOTE: the known_third_party setting is managed by
 # seed-isort-config and should not be modified directly.
 # Any changes made to this setting will be overwritten.
-known_third_party = ["PIL", "bs4", "config", "django", "interface", "lib", "requests"]
+known_third_party = ["bs4", "django", "interface", "requests"]