Merge pull request #43 from th3r00t/newui

Commit changes from newui

README.md

@@ -1,16 +1,18 @@
-# pyShelf 0.5.0
+# pyShelf 0.6.0
 
 <p align="center"><b>Terminal based ebook server. Open source & Lightweight.</b></p>
 <p align="center">Having used Calibre for hosting my eBook collection in the past, I found myself frustrated having to install X on my server, or manage my library externally, Thus I have decided to spin up my own.</p>
 <p align="center"><a href="https://pyshelf.com">https://pyshelf.com</a></p>
 
-![pyShelf 0.5.0 Collection 1](https://github.com/th3r00t/pyShelf/raw/development/preview_050.png)
+![pyShelf 0.6.0 newui](https://github.com/th3r00t/pyShelf/raw/development/src/interface/static/img/pyShelf_frontend_0_2_0.png)
-![pyShelf 0.5.0 Collection 2](https://github.com/th3r00t/pyShelf/raw/development/preview_1_050.png)
 
-<p align="center"><b>Discord [https://discord.gg/H9TbNJS](https://discord.gg/H9TbNJS) | IRC freenode.net @ #pyshelf</b></p>
 
+### You dont need a X server to host a website, or your Movie & Tv collection, so why should you need one to host ebooks?
+<i>Other solutiions require you to have access to an X server to at the very least generate your book database, pyShelf doesnt.We aim to provide a fully featured ebook server with minimal requirements, and no reliance on X whatsoever.</i>
+
+Follow or influence development @ <p align="center"><b> <a href="https://discord.gg/H9TbNJS">Discord</a> |  <a href="https://webchat.freenode.net/#pyshelf">IRC</a> freenode.net @ #pyshelf</b></p>
 ## Current Features
-* Custom Installer -- pre-req installs work on Arch Based Distros Only
+* Custom Installer works only on Arch Based Distros
 * Recursive Scanning
 * Fast database access
 * Django based frontend
@@ -20,36 +22,64 @@
 
 ## Currently Supported Formats
 * epub
+* mobi
 
+
+## 0.6.0 Patch Notes.
+# New Features
+* .mobi Yep mobis are now a thing!
+* Result set ordering
+    * You can now choose to order your results:
+        * Title
+        * Author
+        * Categories
+        * & Tags
+* Reworked UI/UX
+    * More intuitive, less intrusive, & stays out of the way. <i>caveat: I need to rework the placement of the next & previous page controls. While they do remain usable, I intend to have them follow the users</i>
+        position on the page in future releases.
+
+![pyShelf 0.6.0 navbar](https://github.com/th3r00t/pyShelf/raw/development/src/interface/static/img/navbar.png)
+
+* New controls
+    * Sort
+    * Ascending / Descending result set
+    * Display of the result set count, and your current position in the set.
+    * A pop over layer to hold things like
+        * [ ] User login
+        * [ ] Control panel
+        * [ ] Book details
+        * Whatever else :)
 ## Installation Example
 <a href="https://vimeo.com/382292764" target="_blank">pyShelf Installation Video</a>
 
 ## Further Installation & Support Information
 * [SUPPORT.md](https://github.com/th3r00t/pyShelf/blob/development/.github/SUPPORT.md)
 
-## 0.5.0 Patch Notes.
-
 ### Pre-req Dependencies
 * gcc -- This will be installed by the new pre-installer script if its binary is not detected at /usr/bin/gcc
 Users on distros other then Arch should install gcc via their systems package manager prior to
 running the installer.
 * Python3
 * pip
-### New Features
-* Collections
-We are now categorizing your ebooks into collections based on the folder
-structure used to store them. Any folder after the root book folder is now
-considered as a collection.
-#### books/forgotten realms/ -> Forgotten Realms Collection.
-#### books/Dune/Prelude To Dune -> Dune, & Preluse To Dune Collections.
 
-In addition to the work on the collection system, a good deal of time was spent
+# Installation
-on the installer, and the concept of having an installer in the first place.
+This project is currently targeted towards Network Administrators, and other home
-
-I mainly wanted to make this project for Network Administrators, and other home
 enthusiasts whom I assume will know how to setup a Django app, and a
-Postgres server. Beyond that theres nothing the user has to do to make the
+Postgres server.
-system work...
+
+Once your environment is ready very little is required to get the system up and running
+* From the main directory
+    * setup configurations as discussed in [SUPPORT.md](https://github.com/th3r00t/pyShelf/blob/development/.github/SUPPORT.md)
+    * `pip install -r requirments.txt`
+    * `cd src`
+    * `python manage.py migrate`
+    * `cd ..`
+    * `./importbooks`
+    * `./makecollections`
+* Browse to the site as defined in your apache | nginx config
+
+## Included installer
+<a href="https://vimeo.com/382292764" target="_blank">pyShelf Installation Video</a>
 
 The installer will only run correctly on arch based distros. This could be
 easily rectified to include other package managers, Members of the community
@@ -61,6 +91,12 @@ installation already present in the source now, however it is not complete and
 should not be relied upon to be present in future releases unless completed by
 a member of the community,
 
+The installer will walk you through all the configurations required by pyShelf to
+run if you are running on Arch linux.
+
+## Further Installation & Support Information
+* [SUPPORT.md](https://github.com/th3r00t/pyShelf/blob/development/.github/SUPPORT.md)
+
 ## Development
 
 * [`pre-commit`](https://pre-commit.com/)
@@ -90,7 +126,7 @@ Running via the Django test server might be possible, albeit not recomended.
 #### Improved cover image storage, and acquisition.
 #### OPDS Support
 #### Support for other formats
-- [ ] .mobi
+- [x] .mobi
 - [ ] .pdf
 - [ ] .cbz
 - [ ] .zip (Zipped book folders, is this a new idea? (Consider storing your library folders zipped and retrieving a book on demand))

config.json

@@ -1 +1 @@
-{"TITLE": "pyShelf E-Book Server", "VERSION": "0.5.0", "BOOKPATH": "/srv/Books", "DB_HOST": "localhost", "DB_PORT": "5432", "DATABASE": "pyshelf", "USER": "pyshelf", "PASSWORD": "pyshelf", "BOOKSHELF": "data/shelf.json", "ALLOWED_HOSTS": "*", "hostname": "localhost", "webport": "8000", "wsgiport": "8001"}
+{"TITLE": "pyShelf E-Book Server", "VERSION": "0.6.0", "BOOKPATH": "", "DB_HOST": "localhost", "DB_PORT": "5432", "DATABASE": "pyshelf", "USER": "pyshelf", "PASSWORD": "pyshelf", "BOOKSHELF": "data/shelf.json", "ALLOWED_HOSTS": "*", "hostname": "localhost", "webport": "8000", "wsgiport": "8001"}

requirements.txt

@@ -18,3 +18,5 @@ prompt_toolkit
 psutil
 pyfiglet
 mobi-python
+pudb
+jsonpickle

src/backend/lib/library.py

@@ -26,9 +26,10 @@ class Catalogue:
         self.opf_regx = re.compile(r"\.opf")
         self.cover_regx = re.compile(r"\.jpg|\.jpeg|\.png|\.bmp|\.gif")
         self.html_regx = re.compile(r"\.html")
+        self.title_sanitization_regx = re.compile(r"^(Book )+[0-9]*")
+        self.title_sanitization_lvl2_regx = re.compile(r"^(Book )+[0-9]*\W+(-)")
         self.root_dir = config.root
         self.book_folder = config.book_path
-        # self.book_shelf = config.book_shelf
         self.books = None
         self.db_pointer = config.catalogue_db
         self.config = config
@@ -70,6 +71,7 @@ class Catalogue:
         """
 
     def process_by_filetype(self, book):
+        print(str(book), end='\r', flush=True)
         if book.endswith(".epub"):
             epub = self.process_epub(book)
             return self.extract_metadata_epub(epub)
@@ -108,6 +110,11 @@ class Catalogue:
                 title = book["path"].split("/")[-1].rsplit(".", 1)[0]
             else:
                 title = title.contents[0]
+            if re.match(self.title_sanitization_regx, title):
+                if re.match(self.title_sanitization_lvl2_regx, title):
+                    title = re.split(r"-+\W", title)[1]
+                else: title = re.split(self.title_sanitization_regx, title)[2]
+
             author = soup.find("dc:creator")
             if author is not None:
                 author = author.contents[0]
@@ -116,11 +123,57 @@ class Catalogue:
             except IndexError:
                 # cover = self.extract_cover_html(book_zip, book)
                 cover = DuckDuckGo().image_result(title)
-            book_details = [title, author, cover, book["path"]]
+            try:
+                description = self.stripTags(soup.find("dc:description").text)
+            except AttributeError:
+                description = None
+            try:
+                identifier = self.stripTags(soup.find("dc:identifier").text)
+            except AttributeError:
+                identifier = None
+            try:
+                publisher = self.stripTags(soup.find("dc:publisher").text)
+            except AttributeError:
+                publisher = None
+            try:
+                date = self.stripTags(soup.find("dc:date").text)
+            except AttributeError:
+                date = None
+            try:
+                rights = self.stripTags(soup.find("dc:rights").text)
+            except AttributeError:
+                rights = None
+            try:
+                tags = soup.find_all("dc:subject")
+            except AttributeError:
+                tags = None
+            ftags = None
+            if tags is not None:
+                for tag in tags:
+                    if ftags is None:
+                        ftags = tag.text
+                    else:
+                        ftags = ftags + "," + tag.text
+            book_details = [
+                title,
+                author,
+                cover,
+                book["path"],
+                description,
+                identifier,
+                publisher,
+                date,
+                rights,
+                ftags,
+            ]
         return book_details
 
     @staticmethod
-    def extract_metadata_mobi(book):
+    def stripTags(source):
+        p = re.compile(r"<.*?>")
+        return p.sub("", source)
+
+    def extract_metadata_mobi(self, book):
         book = Mobi(book)
         book.parse()
         try:
@@ -129,9 +182,43 @@ class Catalogue:
             cover_image = None
         title = book.title().decode("utf-8")
         author = book.author().decode("utf-8")
-        breakpoint()
+        book_config = book.config
-        # TODO some files are still passing encoded data for author.
+        try:
-        return [title, author, cover_image, book.f.name]
+            description = self.stripTags(book_config['exth']['records'][103].decode("utf-8"))
+        except KeyError:
+            description = None
+        try:
+            identifier = book_config['exth']['records'][104].decode("utf-8")
+        except KeyError:
+            identifier = None
+        try:
+            publisher = book_config['exth']['records'][101].decode("utf-8")
+        except KeyError:
+            publisher = None
+        date = None
+        rights = None
+        try:
+            ftags = book_config['exth']['records'][105].decode("utf-8")
+            if ":" in ftags:
+                ftags = ftags.replace(":", ",")
+            elif ";" in ftags:
+                ftags = ftags.replace(";", ",")
+            # elif re.search(r"\s", ftags):  # Must be final assignment to avoid spliting on multiple delimeters
+            #    ftags = ftags.replace(" ", ",")
+        except KeyError:
+            ftags = None
+        return [
+            title,
+            author,
+            cover_image,
+            book.f.name,
+            description,
+            identifier,
+            publisher,
+            date,
+            rights,
+            ftags,
+        ]
 
     def extract_content(self, book_zip, book):
         """

src/backend/lib/storage.py

@@ -53,7 +53,7 @@ class Storage:
         Insert book in database
         :returns: True if succeeds False if not
         """
-        q = "INSERT INTO books (title, author, cover, progress, file_name, pages) values (%s, %s, %s, 0, %s, 0);"
+        q = "INSERT INTO books (title, author, cover, progress, file_name, pages, description, identifier, publisher, rights, tags) values (%s, %s, %s, 0, %s, 0, %s, %s, %s, %s, %s);"
         try:
             try:
                 cover_image = book[2].data
@@ -61,11 +61,27 @@ class Storage:
                 cover_image = book[2]
             if not book[2]:  # If cover image is missing unset entry
                 cover_image = None
-            self.cursor.execute(q, (book[0], book[1], cover_image, book[3]))
+            self.cursor.execute(
+                q,
+                (
+                    book[0],  # title
+                    book[1],  # author
+                    cover_image,
+                    book[3],  # file
+                    book[4],  # descr
+                    book[5],  # ident
+                    book[6],  # publisher
+                    # book[7],  # date  # TODO: set import time
+                    book[8],  # rights
+                    book[9],  # tags
+                ),
+            )
             return True
         except Exception as e:
-            print(e)
+            if e.pgcode == '22007':  # psycopg2's error code for invalid date
-            return False
+                book[7] = psycopg2.Date(int(book[7]), 1, 1)
+                self.insert_book(book)
+            raise e
 
     def book_paths_list(self):
         """

src/frontend/settings.py

@@ -101,6 +101,10 @@ DATABASES = {
         "PASSWORD": CONFIG.password,
     }
 }
+# Session
+# Uncomment below to enable sessions management by a memcache server
+# https://docs.djangoproject.com/en/3.0/topics/http/sessions/
+# SESSION_ENGINE = "django.contrib.sessions.backends.cached_db"
 
 # Password validation
 # https://docs.djangoproject.com/en/2.2/ref/settings/#auth-password-validators

src/frontend/urls.py

@@ -21,17 +21,20 @@ from interface import views
 urlpatterns = [
     path("admin/", admin.site.urls),
     path("", views.index, name="index"),
+    path("home", views.home, name="index"),
+    path("sort/<_order>", views.index, name="index"),
     path("download/<pk>", views.download, name="download"),
+    path("favorite/<pk>", views.favorite, name="favorite"),
+    path("share/<pk>", views.share, name="share"),
+    path("share/<pk>", views.info, name="info"),
     path("prev_page/<bookset>", views.prev_page, name="prev_page"),
     path("next_page/<bookset>", views.next_page, name="next_page"),
-    path("search/", views.search, name="search"),
+    path("prev_page/<bookset>/<_order>", views.prev_page, name="prev_page"),
-    path("search/<query>", views.search, name="search"),
+    path("next_page/<bookset>/<_order>", views.next_page, name="next_page"),
-    path("search/<query>/<_set>", views.search, name="search"),
+    path("search/", views.index, name="search"),
-    path(
+    path("search/<query>", views.index, name="search"),
-        "show_collection/<_collection>/<_colset>",
+    path("search/<query>/<_set>", views.index, name="search"),
-        views.show_collection,
+    path("show_collection/<_collection>/<_colset>", views.show_collection, name="show_collection",),
-        name="show_collection",
-    ),
 ]
 if settings.DEBUG:
     import debug_toolbar

src/interface/migrations/0005_navigation.py

@@ -0,0 +1,35 @@
+# Generated by Django 3.0.4 on 2020-06-10 05:07
+
+from django.db import migrations, models
+
+
+class Migration(migrations.Migration):
+
+    dependencies = [
+        ("interface", "0004_collections"),
+    ]
+
+    operations = [
+        migrations.CreateModel(
+            name="Navigation",
+            fields=[
+                (
+                    "id",
+                    models.AutoField(
+                        auto_created=True,
+                        primary_key=True,
+                        serialize=False,
+                        verbose_name="ID",
+                    ),
+                ),
+                ("title", models.CharField(max_length=255)),
+                ("link", models.CharField(max_length=255, null=True)),
+                ("category", models.CharField(max_length=255, null=True)),
+                ("parent_id", models.IntegerField(null=True)),
+                ("alt", models.CharField(max_length=255, null=True)),
+                ("type", models.IntegerField(null=True)),
+                ("socket", models.CharField(max_length=255)),
+            ],
+            options={"db_table": "navigation",},
+        ),
+    ]

src/interface/migrations/0006_auto_20200617_2333.py

@@ -0,0 +1,69 @@
+# Generated by Django 3.0.7 on 2020-06-17 23:33
+
+from django.db import migrations, models
+
+
+class Migration(migrations.Migration):
+
+    dependencies = [
+        ('interface', '0005_navigation'),
+    ]
+
+    operations = [
+        migrations.CreateModel(
+            name='Users',
+            fields=[
+                ('id', models.AutoField(auto_created=True, primary_key=True, serialize=False, verbose_name='ID')),
+                ('uname', models.CharField(max_length=255)),
+                ('fname', models.CharField(max_length=255, null=True)),
+                ('lname', models.CharField(max_length=255, null=True)),
+                ('email', models.CharField(max_length=255, null=True)),
+                ('password', models.CharField(max_length=255, null=True)),
+                ('ulvl', models.IntegerField(null=True)),
+            ],
+            options={
+                'db_table': 'users',
+            },
+        ),
+        migrations.AddField(
+            model_name='books',
+            name='date',
+            field=models.DateField(null=True),
+        ),
+        migrations.AddField(
+            model_name='books',
+            name='description',
+            field=models.TextField(null=True),
+        ),
+        migrations.AddField(
+            model_name='books',
+            name='identifier',
+            field=models.CharField(max_length=255, null=True),
+        ),
+        migrations.AddField(
+            model_name='books',
+            name='publisher',
+            field=models.CharField(max_length=266, null=True),
+        ),
+        migrations.AddField(
+            model_name='books',
+            name='rights',
+            field=models.CharField(max_length=255, null=True),
+        ),
+        migrations.AddField(
+            model_name='books',
+            name='tags',
+            field=models.CharField(max_length=255, null=True),
+        ),
+        migrations.CreateModel(
+            name='Favorites',
+            fields=[
+                ('id', models.AutoField(auto_created=True, primary_key=True, serialize=False, verbose_name='ID')),
+                ('favorite', models.ManyToManyField(to='interface.Books')),
+                ('uname', models.ManyToManyField(to='interface.Users')),
+            ],
+            options={
+                'db_table': 'favorites',
+            },
+        ),
+    ]

src/interface/models.py

@@ -29,17 +29,22 @@ class Books(models.Model):
     pages = models.IntegerField(null=True)
     progress = models.IntegerField(null=True)
     file_name = models.CharField(max_length=255, null=False)
+    description = models.TextField(null=True)
+    identifier = models.CharField(max_length=255, null=True)
+    publisher = models.CharField(max_length=266, null=True)
+    date = models.DateField(null=True)
+    rights = models.CharField(max_length=255, null=True)
+    tags = models.CharField(max_length=255, null=True)
 
     def generic_search(self, query):
         try:
             results = Books.objects.annotate(
-                search=SearchVector("title", "file_name", "author"),
+                search=SearchVector("title", "file_name", "author","tags"),
             ).filter(search=query)
         except Exception as e:
             raise
         return results
 
-
 class Collections(models.Model):
     class Meta:
         db_table = "collections"
@@ -62,3 +67,100 @@ class Collections(models.Model):
         except Exception as e:
             raise
         return results
+
+
+class Navigation(models.Model):
+    """
+    pyShelfs Navigation Database class
+    :param title: Link Text
+    :param link: Link link :)
+    :param category: Where in the nav tree do I belong
+    :param parent_id: This link is a sub link of link with id of me
+    :param alt: Alternate text of link
+    :param type: Web link, or Socket link which will be expected to act on \
+    the link, and the action defined in socket
+    :param socket: if a Socket link define socket here
+    """
+
+    class Meta:
+        db_table = "navigation"
+
+    def __str__(self):
+        return self.title
+
+    title = models.CharField(max_length=255)
+    link = models.CharField(max_length=255, null=True)
+    category = models.CharField(max_length=255, null=True)
+    parent_id = models.IntegerField(null=True, editable=True)
+    alt = models.CharField(max_length=255, null=True)
+    type = models.IntegerField(null=True)
+    socket = models.CharField(max_length=255, null=False)
+
+    def generic_search(self, query):
+        try:
+            results = Navigation.objects.annotate(
+                search=SearchVector("title", "parent_id", "category"),
+            ).filter(search=query)
+        except Exception as e:
+            raise
+        return results
+
+
+class Users(models.Model):
+    """
+    pyShelfs User Database class
+    :param uname: User Name
+    :param fname: First Name
+    :param lname: Last Name
+    :param email: User Email Address
+    :param password: User Password
+    :param ulvl: User Level
+    """
+
+    class Meta:
+        db_table = "users"
+
+    def __str__(self):
+        return self.title
+
+    uname = models.CharField(max_length=255)
+    fname = models.CharField(max_length=255, null=True)
+    lname = models.CharField(max_length=255, null=True)
+    email = models.CharField(max_length=255, null=True, editable=True)
+    password = models.CharField(max_length=255, null=True)
+    ulvl = models.IntegerField(null=True)
+
+    def generic_search(self, query):
+        try:
+            results = Users.objects.annotate(
+                search=SearchVector("uname", "email", "lname"),
+            ).filter(search=query)
+        except Exception as e:
+            raise
+        return results
+
+
+class Favorites(models.Model):
+    """
+    pyShelfs User Database class
+    :param uname: User Name
+    :param fname: First Name
+    """
+
+    class Meta:
+        db_table = "favorites"
+
+    def __str__(self):
+        return self.title
+
+    favorite = models.ManyToManyField(Books)
+    uname = models.ManyToManyField(Users)
+
+    def generic_search(self, query):
+        try:
+            results = Favorites.objects.annotate(search=SearchVector("uname"),).filter(
+                search=query
+            )
+        except Exception as e:
+            raise
+        return results