th3r00t/pyShelf
1
0
Fork 0
mirror of https://github.com/th3r00t/pyShelf.git synced 2026-04-28 01:59:35 -04:00
Code Issues Packages Projects Releases Wiki Activity

Refactoring collections handler

Browse Source
This commit is contained in:
th3r00t
2023-03-20 12:12:55 -04:00
parent 11c0c89624
commit 3e89273cbc
10 changed files with 289 additions and 49 deletions
Download Patch File Download Diff File Expand all files Collapse all files

42
docs/Automated Collections.html Normal file
View File

@@ -0,0 +1,42 @@
<!DOCTYPE html>
<html>
<head>
<link rel="Stylesheet" type="text/css" href="style.css">
<link rel="alternate" type="application/rss+xml" title="RSS" href="rss.xml">
<title>Automated Collections</title>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
<meta name="viewport" content="width=device-width, initial-scale=1">
</head>
<body>
<div id="Automated Collection Management System"><h1 id="Automated Collection Management System" class="header"><a href="#Automated Collection Management System">Automated Collection Management System</a></h1></div>
<p>
The collection management system needs rewritten from the ground up.
The model should look something like this.
</p>
<ul>
<li class="done0">
<span class="todo">TODO</span> :: Refactor collections algorithm. #8f46262c
<pre python>
Collection {
id: int
name: String
books: list(Book)
}
</pre>
</ul>
<p>
it is likely the book object will need rewritten as well.
</p>
<pre python>
Book {
id: int
title: String
# other book metadata
collection: list(Collection.id)
}
</pre>
</body>
</html>

14
docs/Book Scraping System.html
View File

@@ -29,25 +29,25 @@ sub-folders in your collection.
<div id="Book Scraping System-Scrapers"><h2 id="Scrapers" class="header"><a href="#Book Scraping System-Scrapers">Scrapers</a></h2></div> <div id="Book Scraping System-Scrapers"><h2 id="Scrapers" class="header"><a href="#Book Scraping System-Scrapers">Scrapers</a></h2></div>
<ul> <ul>
<li class="done4"> <li class="done4">
epub #75f00edf epub #2899a8e9
<li class="done4"> <li class="done4">
mobi #2fe4b161 mobi #ec035720
<li class="done4"> <li class="done4">
pdf #ffa7e7f0 pdf #05875e64
<li class="done0"> <li class="done0">
cbz #64b5da95 cbz #4a513e39
</ul> </ul>
<div id="Book Scraping System-Collections"><h2 id="Collections" class="header"><a href="#Book Scraping System-Collections">Collections</a></h2></div> <div id="Book Scraping System-Collections"><h2 id="Collections" class="header"><a href="#Book Scraping System-Collections">Collections</a></h2></div>
<ul> <ul>
<li class="done0"> <li class="done0">
Manual Collections #2e7e6fcf Manual Collections #b07156f4
<li class="done4"> <li class="done0">
Automated Collections #81db675a <a href="Automated Collections.html">Automated Collections</a> #f258c1f8
</ul> </ul>
<div id="Book Scraping System-State"><h2 id="State" class="header"><a href="#Book Scraping System-State">State</a></h2></div> <div id="Book Scraping System-State"><h2 id="State" class="header"><a href="#Book Scraping System-State">State</a></h2></div>

113
docs/Client.html Normal file
View File

@@ -0,0 +1,113 @@
<!DOCTYPE html>
<html>
<head>
<link rel="Stylesheet" type="text/css" href="style.css">
<link rel="alternate" type="application/rss+xml" title="RSS" href="rss.xml">
<title>Client</title>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
<meta name="viewport" content="width=device-width, initial-scale=1">
</head>
<body>
<p>
<a href="index.html">Home</a> <a href="TOC.html">TOC</a> <a href="https://github.com/th3r00t/pyShelf.git">github</a>
</p>
<div id="Client"><h1 id="Client" class="header"><a href="#Client">Client</a></h1></div>
<p>
<span id="Client-frontend"></span><span class="tag" id="frontend">frontend</span> <span id="Client-ui"></span><span class="tag" id="ui">ui</span>
</p>
<div id="Client-Dependency Management"><h2 id="Dependency Management" class="header"><a href="#Client-Dependency Management">Dependency Management</a></h2></div>
<p>
<span id="Client-Dependency Management-dependency"></span><span class="tag" id="dependency">dependency</span>
</p>
<ul>
<li class="done4">
npm #c5f8d347
<li class="done4">
sass #8f43ab4c
<li class="done0">
pip #14ee256e
</ul>
<div id="Client-[[https://create.t3.gg|T3.gg]]"><h2 id="[[https://create.t3.gg|T3.gg]]" class="header"><a href="#Client-[[https://create.t3.gg|T3.gg]]"><a href="https://create.t3.gg">T3.gg</a></a></h2></div>
<div id="Client-[[https://create.t3.gg|T3.gg]]-User System"><h3 id="User System" class="header"><a href="#Client-[[https://create.t3.gg|T3.gg]]-User System">User System</a></h3></div>
<ul>
<li>
NextAuthjs
<li>
Authentication Providers
<ul>
<li>
Discord
Required: Client ID, Client Secret
<li>
Github
Required: Client ID, Client Secret, only one callback per app.
<img src="https://next-auth.js.org/providers/github" alt="NextAuthjs Github" />
</ul>
</ul>
<div id="Client-[[https://create.t3.gg|T3.gg]]-User Experience"><h3 id="User Experience" class="header"><a href="#Client-[[https://create.t3.gg|T3.gg]]-User Experience">User Experience</a></h3></div>
<p>
<span id="Client-[[https://create.t3.gg|T3.gg]]-User Experience-ux"></span><span class="tag" id="ux">ux</span>
</p>
<ul>
<li class="done0">
Favorites #0fc67f94
<li class="done0">
Permissions / Roles #9c4f279e
<li class="done0">
User Profile #289e0fe0
<li class="done0">
User Settings #51955b33
</ul>
<div id="Client-[[https://create.t3.gg|T3.gg]]-Data Management"><h3 id="Data Management" class="header"><a href="#Client-[[https://create.t3.gg|T3.gg]]-Data Management">Data Management</a></h3></div>
<p>
<span id="Client-[[https://create.t3.gg|T3.gg]]-Data Management-data"></span><span class="tag" id="data">data</span>
</p>
<ul>
<li class="done0">
Book Management System #92e523d2
<ul>
<li class="done0">
add #bda9a0f3
<li class="done0">
remove #4b34e931
<li class="done0">
update #eb959340
<li class="done0">
access level #b00795a4
<li class="done0">
ownership #8c80edc5
<li class="done0">
attach #8acb21a9
<li class="done0">
detach #2e263616
</ul>
</ul>
</body>
</html>

2
docs/index.html
View File

@@ -108,7 +108,7 @@ Filesystem-io
<a href="REST API.html">REST API</a> #c7bc51c5 <a href="REST API.html">REST API</a> #c7bc51c5
<li class="done0"> <li class="done0">
<a href="Frontend.html">Frontend</a> #a76c1038 <a href="Client.html">Client</a> #c68b0664
</ul> </ul>
<div id="pyShelf | project:pyshelf-Development"><h2 id="Development" class="header"><a href="#pyShelf | project:pyshelf-Development">Development</a></h2></div> <div id="pyShelf | project:pyshelf-Development"><h2 id="Development" class="header"><a href="#pyShelf | project:pyshelf-Development">Development</a></h2></div>

15
src/backend/lib/models.py vendored
View File

@@ -6,22 +6,20 @@ import datetime
timestamp = Annotated[ timestamp = Annotated[
datetime.datetime, datetime.datetime,
mapped_column(nullable=False, server_default=func.CURRENT_TIMESTAMP()) mapped_column(nullable=False, server_default=func.CURRENT_TIMESTAMP()),
] ]
class Base(DeclarativeBase): class Base(DeclarativeBase):
"""Base class for all models.""" """Base class for all models."""
pass
class Book(Base): class Book(Base):
"""Book model.""" """Book model."""
__tablename__ = "books" __tablename__ = "Book"
book_id: Mapped[int] = mapped_column(primary_key=True, nullable=False) id: Mapped[int] = mapped_column(primary_key=True, nullable=False)
title: Mapped[str] title: Mapped[str]
author: Mapped[Optional[str]] author: Mapped[Optional[str]]
categories: Mapped[Optional[str]] categories: Mapped[Optional[str]]
@@ -37,12 +35,11 @@ class Book(Base):
publisher: Mapped[Optional[str]] publisher: Mapped[Optional[str]]
class Collection(Base): class Collection(Base):
"""Collection model.""" """Collection model."""
__tablename__ = "collections" __tablename__ = "Collection"
id: Mapped[int] = mapped_column(primary_key=True)
collection: Mapped[str] collection: Mapped[str]
book_id: Mapped[int] = mapped_column(ForeignKey(Book.book_id)) book_id: Mapped[int] = mapped_column(ForeignKey(Book.id))
collection_id: Mapped[int] = mapped_column(primary_key=True)

76
src/backend/lib/storage.py vendored
View File

@@ -1,6 +1,5 @@
"""Pyshelf's Main Storage Class.""" """Pyshelf's Main Storage Class."""
import re import re
import os
from sqlalchemy import create_engine, select from sqlalchemy import create_engine, select
from sqlalchemy.orm import Session from sqlalchemy.orm import Session
@@ -36,8 +35,7 @@ class Storage:
self.password = self.config.password self.password = self.config.password
self.db_host = self.config.db_host self.db_host = self.config.db_host
self.db_port = self.config.db_port self.db_port = self.config.db_port
self.engine = create_engine(self.get_connection_string(), self.engine = create_engine(self.get_connection_string(), pool_pre_ping=True)
pool_pre_ping=True)
def get_connection_string(self): def get_connection_string(self):
"""Get connection string. """Get connection string.
@@ -49,7 +47,7 @@ class Storage:
str : sqlalchemy Connection String str : sqlalchemy Connection String
""" """
if self.config.db_engine == "sqlite": if self.config.db_engine == "sqlite":
return f"sqlite:////{self.config.root}/pyshelf.db" return f"sqlite:////{self.config.root}/pyshelf.sqlite3"
elif self.config.db_engine == "psql": elif self.config.db_engine == "psql":
return f"postgresql://{self.user}:{self.password}\ return f"postgresql://{self.user}:{self.password}\
@{self.db_host}:{self.db_port}/{self.sql}" @{self.db_host}:{self.db_port}/{self.sql}"
@@ -86,6 +84,8 @@ class Storage:
cover_image = None cover_image = None
if not book[1]: if not book[1]:
pass pass
# breakpoint()
self.parse_collections_from_path(book)
_book = Book( _book = Book(
title=book[0], title=book[0],
author=book[1], author=book[1],
@@ -117,6 +117,36 @@ class Storage:
session.close() session.close()
return _result return _result
def parse_collections_from_path(self, book: dict()) -> list():
"""Parse book path's to determine common folder structure.
Stores collections based on shared paths.
Parameters
----------
book : dict()
Book object to parse.
Returns
-------
collections : list()
List of collections.
"""
collections = []
title_regx = re.compile(r"^[0-9][0-9]*|-|\ \B")
_pathing = book[3].split(self.config.book_path + "/")[1].split("/")
try:
_pathing.pop(0)
_pathing.pop(-1)
except IndexError:
pass
for _p in _pathing:
_s = _p.replace("'", "")
_x = re.sub(title_regx, "", _s)
_s = _x.strip()
collections.append(_s)
return collections
def make_collections(self): def make_collections(self):
"""Parse book path's to determine common folder structure. """Parse book path's to determine common folder structure.
@@ -126,7 +156,7 @@ class Storage:
self.config.logger.info("Making collections.") self.config.logger.info("Making collections.")
_title_regx = re.compile(r"^[0-9][0-9]*|-|\ \B") _title_regx = re.compile(r"^[0-9][0-9]*|-|\ \B")
session = Session(self.engine) session = Session(self.engine)
_set = session.execute(select(Book.book_id, Book.file_name)).all() _set = session.execute(select(Book.id, Book.file_name)).all()
if _set.__len__() > 0: if _set.__len__() > 0:
for book in _set: for book in _set:
path = self.config.book_path + "/" path = self.config.book_path + "/"
@@ -143,25 +173,22 @@ class Storage:
_s = _x.strip() _s = _x.strip()
_sess = Session(self.engine) _sess = Session(self.engine)
_q = _sess.execute( _q = _sess.execute(
select(Collection.collection_id).where( select(Collection.id).where(
Collection.collection == _s, Collection.collection == _s,
Collection.book_id == book.book_id, Collection.book_id == book.id,
) )
) )
_sess.close() _sess.close()
if _q.fetchone() is None: if _q.fetchone() is None:
_collection = Collection( _collection = Collection(collection=_s, book_id=book.id)
collection=_s, book_id=book.book_id)
with Session(self.engine) as _sess: with Session(self.engine) as _sess:
try: try:
_sess.add(_collection) _sess.add(_collection)
_sess.commit() _sess.commit()
_sess.close() _sess.close()
self.config.logger.info( self.config.logger.info(f"Collection {_s} added.")
f"Collection {_s} added.")
except Exception as e: except Exception as e:
self.config.logger.error( self.config.logger.error(f"Collection {_s} failed: {e}")
f"Collection {_s} failed: {e}")
_collections.append(_p) _collections.append(_p)
self.config.logger.info("Finished making collections.") self.config.logger.info("Finished making collections.")
@@ -180,21 +207,23 @@ class Storage:
session = Session(self.engine) session = Session(self.engine)
if collection: if collection:
_result = session.execute( _result = session.execute(
select(Book).join(Collection) select(Book)
.where(Collection.collection_id == collection) .join(Collection)
.offset(skip).limit(limit)).all() .where(Collection.id == collection)
.offset(skip)
.limit(limit)
).all()
else: else:
_result = session.execute( _result = session.execute(select(Book).offset(skip).limit(limit)).all()
select(Book).offset(skip).limit(limit)).all()
session.close() session.close()
return _result return _result
def get_book(self, book_id): def get_book(self, id):
"""Get book from database. """Get book from database.
Parameters Parameters
---------- ----------
book_id : int id : int
Book ID to filter by. Book ID to filter by.
Returns Returns
@@ -202,7 +231,7 @@ class Storage:
_result : ScalarResult Object _result : ScalarResult Object
""" """
session = Session(self.engine) session = Session(self.engine)
_result = session.execute(select(Book).where(Book.book_id == book_id)).first() _result = session.execute(select(Book).where(Book.id == id)).first()
session.close() session.close()
return _result return _result
@@ -214,9 +243,6 @@ class Storage:
_result : ScalarResult Object _result : ScalarResult Object
""" """
session = Session(self.engine) session = Session(self.engine)
_result = session.execute( _result = session.execute(select(Collection).join(Book)).all()
select(Collection)
.join(Book)
).all()
session.close() session.close()
return _result return _result

21
wiki/Automated Collections.wiki vendored Normal file
View File

@@ -0,0 +1,21 @@
= Automated Collection Management System =
The collection management system needs rewritten from the ground up.
The model should look something like this.
* [ ] TODO :: Refactor collections algorithm. #8f46262c
{{{python
Collection {
id: int
name: String
books: list(Book)
}
}}}
it is likely the book object will need rewritten as well.
{{{python
Book {
id: int
title: String
# other book metadata
collection: list(Collection.id)
}
}}}

12
wiki/Book Scraping System.wiki vendored
View File

@@ -10,14 +10,14 @@ pyShelf features a recursive scraping algorithm that itterates over all
sub-folders in your collection. sub-folders in your collection.
== Scrapers == == Scrapers ==
* [X] epub #75f00edf * [X] epub #2899a8e9
* [X] mobi #2fe4b161 * [X] mobi #ec035720
* [X] pdf #ffa7e7f0 * [X] pdf #05875e64
* [ ] cbz #64b5da95 * [ ] cbz #4a513e39
== Collections == == Collections ==
* [ ] Manual Collections #2e7e6fcf * [ ] Manual Collections #b07156f4
* [X] Automated Collections #81db675a * [ ] [[Automated Collections]] #f258c1f8
== State == == State ==

41
wiki/Client.wiki vendored Normal file
View File

@@ -0,0 +1,41 @@
[[index|Home]] [[TOC]] [[https://github.com/th3r00t/pyShelf.git|github]]
= Client =
:frontend:ui:
== Dependency Management ==
:dependency:
* [X] npm #c5f8d347
* [X] sass #8f43ab4c
* [ ] pip #14ee256e
== [[https://create.t3.gg|T3.gg]] ==
=== User System ===
- NextAuthjs
- Authentication Providers
- Discord
Required: Client ID, Client Secret
- Github
Required: Client ID, Client Secret, only one callback per app.
{{https://next-auth.js.org/providers/github|NextAuthjs Github}}
=== User Experience ===
:ux:
* [ ] Favorites #0fc67f94
* [ ] Permissions / Roles #9c4f279e
* [ ] User Profile #289e0fe0
* [ ] User Settings #51955b33
=== Data Management ===
:data:
* [ ] Book Management System #92e523d2
* [ ] add #bda9a0f3
* [ ] remove #4b34e931
* [ ] update #eb959340
* [ ] access level #b00795a4
* [ ] ownership #8c80edc5
* [ ] attach #8acb21a9
* [ ] detach #2e263616

2
wiki/index.wiki vendored
View File

@@ -38,7 +38,7 @@ pyShelf supports the following formats:
== TODO == == TODO ==
* [ ] [[Book Scraping System]] #f7edafb1 * [ ] [[Book Scraping System]] #f7edafb1
* [ ] [[REST API]] #c7bc51c5 * [ ] [[REST API]] #c7bc51c5
* [ ] [[Frontend]] #a76c1038 * [ ] [[Client]] #c68b0664
== Development == == Development ==