Funds $0 | Needed $120
Like the community? Enjoy reading manga?
Then click on the small donation banner! Thx!

0%

Author Topic: Novel downloader\converter userscript (Baka-Tsuki, Shosetsuka ni Narou(ncode.syosetu.com), pastebin)  (Read 30497 times)

0 Members and 1 Guest are viewing this topic.

Offline Poligrafowicz

  • J-Hero
  • ***
  • Posts: 454
  • Renommée: 155
  • Gender: Male
    • View Profile
To Poligrafowicz

I think I love you man, please marry me.
Unless you're a girl (which you don't seem to be), I'm not interested.

By the way, Is there a way to get rid of the "Note:" section on the left hand size for HTML mode?

this is because my screen is small so I had to zoom, causing that section and the novel content to overlap.
If it's not a problem for you, you could edit the html file, find the <div id="fixed"> tag and add a hiding style, ie. change it to <div id="fixed" style="display:none">.

It should be possible to resize the note text area (lower right corner) and the new size should be actually saved in the cookies, but if even the <b>Notes:</b> sign is too big, I might add some simple responsive css to hide it on certain resolutions. How wide is your screen in pixels?

Offline fickle

  • J-Starter
  • *
  • Posts: 16
  • Renommée: 0
  • Gender: Male
    • View Profile
To Poligrafowicz

I think I love you man, please marry me.


By the way, Is there a way to get rid of the "Note:" section on the left hand size for HTML mode?

this is because my screen is small so I had to zoom, causing that section and the novel content to overlap.

Offline Poligrafowicz

  • J-Hero
  • ***
  • Posts: 454
  • Renommée: 155
  • Gender: Male
    • View Profile
Updated Baka-Tsuki downloader to allow downloading from Wayback machine (note wayback seems to be missing css for BT for some reason and the download link may appear at somewhere the bottom of the page instead of the top menu). Also added https variants of urls to the script.

Sorry, please read the forum rules to see why you can't view spoilers and why you can't post in this forum section. Thank you!

Offline Poligrafowicz

  • J-Hero
  • ***
  • Posts: 454
  • Renommée: 155
  • Gender: Male
    • View Profile
Updated chapter splitter, now it can split into volumes too (currently there might be duplicated images in subfolders; I'll improve it later)
DEPRECIATED
Sorry, please read the forum rules to see why you can't view spoilers and why you can't post in this forum section. Thank you!
EXE MD5: a6f5f8e7dce52df4440572d2de34880c
I'm also considering adding base64 images as an option (pro: all resources in a single html file; con: huge files)

Offline Poligrafowicz

  • J-Hero
  • ***
  • Posts: 454
  • Renommée: 155
  • Gender: Male
    • View Profile
Can't see the HTML or HTML (Ruby) button at all when logged in.

Try this version
DEPRECIATED
Sorry, please read the forum rules to see why you can't view spoilers and why you can't post in this forum section. Thank you!

Offline XerxesDraconis

  • J-Regular
  • ***
  • Posts: 203
  • Renommée: 97
  • Gender: Male
    • View Profile
Can't see the HTML or HTML (Ruby) button at all when logged in.

Offline azurehyaline

  • J-Regular
  • ***
  • Posts: 222
  • Renommée: 49
    • View Profile
Small fix for those using https instead of http:

Sorry, please read the forum rules to see why you can't view spoilers and why you can't post in this forum section. Thank you!

Offline Poligrafowicz

  • J-Hero
  • ***
  • Posts: 454
  • Renommée: 155
  • Gender: Male
    • View Profile
want to ask, is there application to change HTML to PDF?

Google for a PDF printer.

---------------------------

Depreciated
Updated the Chapter Splitter. The new version should keep images after splitting.
MD5 of the binary: 9f350ef5a7499a9f5d0beda9577c55ac
Sorry, please read the forum rules to see why you can't view spoilers and why you can't post in this forum section. Thank you!

Offline Nightmare619

  • J-Starter
  • *
  • Posts: 12
  • Renommée: 0
  • Gender: Male
    • View Profile
want to ask, is there application to change HTML to PDF?

Offline Poligrafowicz

  • J-Hero
  • ***
  • Posts: 454
  • Renommée: 155
  • Gender: Male
    • View Profile
Another update for Narou downloader. This time I tried to increase downloader's compatibility with older versions of narou (you might encounter those on wayback machine).
DEPRECIATED
Sorry, please read the forum rules to see why you can't view spoilers and why you can't post in this forum section. Thank you!

Offline kipsss07

  • J-Starter
  • *
  • Posts: 15
  • Renommée: 0
  • Gender: Male
    • View Profile

Offline Poligrafowicz

  • J-Hero
  • ***
  • Posts: 454
  • Renommée: 155
  • Gender: Male
    • View Profile
Updated Shosetsuka ni Narou downloader to support downloading from wayback machine (in case a series was removed from Narou).
Depreciated
Sorry, please read the forum rules to see why you can't view spoilers and why you can't post in this forum section. Thank you!

Offline Poligrafowicz

  • J-Hero
  • ***
  • Posts: 454
  • Renommée: 155
  • Gender: Male
    • View Profile
Shousetsuka ni narou downloader was update. The current version has the ability to retain rubygana. Reasoning for this was, complete lack of furigana in the text version was starting to get problematic when reading and my means of reading text containing rubygana have increased. Since rubygana is the actual standard of using furigana in html, I figured, I might as well allow for it to be left as is.
Depreciated
Sorry, please read the forum rules to see why you can't view spoilers and why you can't post in this forum section. Thank you!

Offline vsdgniuz

  • J-Starter
  • *
  • Posts: 2
  • Renommée: 0
    • View Profile
Excellent work on Baka-tsuki script. Glad I searched google as I was thinking of writing such a script myself. Great work. Highly appreciated.
PS - I think this script needs to be posted on BT forum. Am using this with tampermonkey on chrome.

Edit - Create ebook feature on BT has not been working from November, thus required a substitute and this worked perfectly for me :)

Offline Poligrafowicz

  • J-Hero
  • ***
  • Posts: 454
  • Renommée: 155
  • Gender: Male
    • View Profile
I guess scriptish handles scripts on sites differently. I only used jquery on sites which already used it. Greasemonkey allows doing that.

Offline shadowlight168

  • J-Starter
  • *
  • Posts: 13
  • Renommée: 0
  • Gender: Male
    • View Profile
using the scriptish addon i had to add
Code: [Select]
// @require       http://code.jquery.com/jquery-2.1.1.min.jsfor the script to work for me

just posting this incase there is anyone else like me who had trouble getting the script to work.

Offline Poligrafowicz

  • J-Hero
  • ***
  • Posts: 454
  • Renommée: 155
  • Gender: Male
    • View Profile
Updated Pastebin downloader in first post with global checkbox (so you can select\unselect everything now).

Offline Poligrafowicz

  • J-Hero
  • ***
  • Posts: 454
  • Renommée: 155
  • Gender: Male
    • View Profile
Updated version of chapter splitter:
Depreciated
Sorry, please read the forum rules to see why you can't view spoilers and why you can't post in this forum section. Thank you!
EXE MD5: 4196137606150d36868177c810e5b869

What's changed:
Chapter numbers have now whooping 4 digits, so if your device isn't smart enough to sort them without the extra zeros than you'll like this (unfortunately JadeReader has this kind of problem hence the update).

Offline Poligrafowicz

  • J-Hero
  • ***
  • Posts: 454
  • Renommée: 155
  • Gender: Male
    • View Profile
Re: Novel downloader\converter userscript (Baka-Tsuki, ncode.syosetu.com)
« Reply #14 on: September 24, 2014, 01:11:06 PM »
Depreciated
I bring you a Pastebin downloader:
Sorry, please read the forum rules to see why you can't view spoilers and why you can't post in this forum section. Thank you!

How does it work:
1. Go to someone's Pastebin
2. Select the chapters you want to download (though there is no mass unselect and everything is selected by default)
3. Click the "download" button in the second top menu (the one that has "sign up")
4. You get all the text dumpend into a single html page with a table of contents at the top
5. Save the html file
6. Convert it using Sigil to epub (or using other tool to other format or just keep it as html)

Why?
Because someone is posting Tate no Yuusha translation through Pastebin

Offline maxsteele77

  • J-Starter
  • *
  • Posts: 11
  • Renommée: 1
    • View Profile
Re: Novel downloader\converter userscript (Baka-Tsuki, ncode.syosetu.com)
« Reply #13 on: September 15, 2014, 09:04:15 PM »
Thanks a bunch for the Baka-tsuki script.

Their .epub generator doesn't always work properly so sometimes I would just save the webpage as an html but that would only save the image thumbnails so I'd have to manually open then save the full size images. Your script will save me an absolute shit ton of time and is easy as fuck to use.

Thanks again :)

Offline Hussbek

  • J-Starter
  • *
  • Posts: 3
  • Renommée: 0
    • View Profile
Re: Novel downloader\converter userscript (Baka-Tsuki, ncode.syosetu.com)
« Reply #12 on: September 11, 2014, 03:21:09 AM »
Good job.s
I mostly read BT novele these days so this is perfect!!!

I also read from that site but i always had to wait for pdfs so this truly helps

Offline Poligrafowicz

  • J-Hero
  • ***
  • Posts: 454
  • Renommée: 155
  • Gender: Male
    • View Profile
Re: Novel downloader\converter userscript (Baka-Tsuki, ncode.syosetu.com)
« Reply #11 on: September 10, 2014, 09:41:47 AM »
Small update to the chapter splitter. I tried to split Tate no Yuusha into chapters and it crashed. Turns out I should have done something to prevent illegal characters from appearing in file names. This update replaces all illegal characters in file and folder names with "_" (underscore).

I also tested what happens if your destination folder already contains files with same names and the app also crashes (can't overwrite existing files and just throws an exception), so it is advised to use an empty folder for the splitting.

Depreciated
Sorry, please read the forum rules to see why you can't view spoilers and why you can't post in this forum section. Thank you!
Note: EXE file should have the following MD5 8ff3f399eab54edeedadeafa2ca184df.

P.S. Lately I started experimenting with node-webkit and it looks like an ideal platform for writing downloaders and small tools like this chapter splitter. I'm thinking of patching up a more advanced, standalone  ncode downloader.

Offline fafati

  • J-Starter
  • *
  • Posts: 2
  • Renommée: 0
  • Gender: Female
    • View Profile
Re: Novel downloader\converter userscript (Baka-Tsuki, ncode.syosetu.com)
« Reply #10 on: September 08, 2014, 01:59:04 AM »
thank you

Offline Poligrafowicz

  • J-Hero
  • ***
  • Posts: 454
  • Renommée: 155
  • Gender: Male
    • View Profile
Re: Novel downloader\converter userscript (Baka-Tsuki, ncode.syosetu.com)
« Reply #9 on: September 06, 2014, 11:08:42 AM »
Edit: I've actually did a small fix to the ncode user script:
DEPRECIATED
Sorry, please read the forum rules to see why you can't view spoilers and why you can't post in this forum section. Thank you!

In this version the completely redundant new line characters that appear countless times in the file are removed (you can only see them if you open up the output html file in a text editor). Additionally the TXT version no longer contains the style tag in the header. I removed it because the software that I used for reading those didn't support CSS nor the fonts mentioned in it.

-------------------------------------

Not really an update (lately I've been more into using those scripts and reading the results than upgrading them) to the download script, but here is a tool for splitting ncode downloads into chapters.
DEPRECIATED
Sorry, please read the forum rules to see why you can't view spoilers and why you can't post in this forum section. Thank you!

This archive contains both executable file (ChapterSplitter\bin\Release\ChapterSplitter.exe [MD5:b7769f21862ce83f0b0409370f92aecf]*) and source code. If you don't trust me enough to run the executable, you can compile the tool yourself (just make sure to nuget HtmlAgilityPack for the project on your computer). The source code should be simplistic enough to let  yoy verify that the app is harmless.

Anyway, why this app? I've started reading some webnovels on an Android device using Jade Reader (basically a standalone Rikaichan for Android). However Jade Reader has some trouble handling big files and doesn't support anchors. As such I had hard time navigating within the document. Splitting web novel into chapters helped me a lot with loading times and picking the chapters that I want to read (untranslated gaps in Mushoku Tensei BT translation). I did try to improve Jade Reader, but I had really hard time jumping into reverse engineering a Java (a language which I don't know) application, so I gave up and wrote the splitter instead.

How to use:
1. Start the app
2. Press the "Load file..." button (hard to miss)
3. Select the HTML file downloaded from ncode that you want to split (note it has to be the TXT version)
4. Select destination directory (prepare an empty one yourself)
5. ???
6. Profit (novel will be split into chapters; additionally if the web novel had volumes declared, the chapters will be placed in subfolders named after those)

Potential problems: If chapter\volume name contains forbidden characters / ? < > \ : * | " the app might crash (I haven't encountered those while splitting whole Mushoku Tensei into chapters, but some other novels might actually contain those). I haven't tested if the app overwrites existing files or not (if it's the latter, it might crash).

* I checked the file that was uploaded to GE.TT and the MD5 matched with the one on my HDD, but I'm not sure about other hosting services (some might plant spyware\crapware\viruses in exe files).

Edit: Additionally the splitter also removes the style section from the html file. I mean, CSS doesn't work with Jade Reader anyway (in fact it displays the contents of the style tag), so it was pointless keeping it there.

Offline Poligrafowicz

  • J-Hero
  • ***
  • Posts: 454
  • Renommée: 155
  • Gender: Male
    • View Profile
Re: Novel downloader\converter userscript (Baka-Tsuki, ncode.syosetu.com)
« Reply #8 on: August 10, 2014, 01:54:30 PM »
DEPRECIATED
Since I couldn't decide on one furigana mode, I ended up adding them both as well as keeping the faulty old mode and adding the ability to turn off furigana altogether. I also added cookie based notes, so you can make quick vocab notes if you want to (not sure if it works in browsers other than FireFox). This is all of course for the HTML version. The only real change that I made in the TXT version is that dot furigana now turn corresponding kanji to bold. Speaking of which, dot furigana should now style properly in the HTML version (one dot above each character).

Sorry, please read the forum rules to see why you can't view spoilers and why you can't post in this forum section. Thank you!

The only missing feature now are the kanji stats.

Offline Poligrafowicz

  • J-Hero
  • ***
  • Posts: 454
  • Renommée: 155
  • Gender: Male
    • View Profile
Re: Novel downloader\converter userscript (Baka-Tsuki, ncode.syosetu.com)
« Reply #7 on: August 10, 2014, 07:56:59 AM »
I've experimented with fixing overlapping furigana and there are two ways to do this. However I'd like to hear your opinion on the matter.  :megane:

Figure A shows the problem of overlapping furigana. The first way to solve this problem is by mimicing Ruby. Ruby adds spacing between kanji if furigana is wider than the kanji to prevent it from overlapping . You can see this solution on figure B. The other way to solve it is by moving furigana sideways (as shown on figure C). Personally I think the second solution looks better, but it will fail in case of 3 overlapping furigana groups (I wonder if this actually is present in any of the novels) while the first method will handle those fine. What do you think?

Offline Poligrafowicz

  • J-Hero
  • ***
  • Posts: 454
  • Renommée: 155
  • Gender: Male
    • View Profile
Re: Novel downloader\converter userscript (Baka-Tsuki, ncode.syosetu.com)
« Reply #6 on: August 10, 2014, 05:34:51 AM »
DEPRECIATED
Here is a new version of  http://ncode.syosetu.com downloader:
Sorry, please read the forum rules to see why you can't view spoilers and why you can't post in this forum section. Thank you!
If you already installed the previous version in your browser, uninstall it first. This downloader allows you to select which chapters you want to download and allows both conversion to TXT and HTML. Note that the TXT version still contains some basic html formating. Ruby furigana are now stripped from the TXT version. HTML version mimics ncode's styling and converts Ruby furigana to some simple javascript enabled furigana. The thing about Ruby furigana is that it is placed directly in the text which interfers with Rikaichan and selecting text. My furigana solution just places it at the end of document and then positions it in the text which should allow Rikaichan and text selection to work normally. Furigana will work after you save the converted novel and reopen it. Other extras are still in the works (translating jquery to plain javascript). Furigana can sometimes overlap, but I'll try to fix it later.

When using the script on a chapter page, only the current chapter will be saved.

The new version of the script also fixes a minor bug in the TXT converter which would cause some of the original style and classes to retain after conversion.

Another known bug is that if you have images in the novel (some appear to have them), those will have a direct path to the file on your HDD after you save the document in your browser. This is just something you have to be careful about if you change the location of your copy of the novel or share it with others. I might patch up some small tool for fixing that if anyone is interested.

Offline Poligrafowicz

  • J-Hero
  • ***
  • Posts: 454
  • Renommée: 155
  • Gender: Male
    • View Profile
Re: Baka-Tsuki to plain HTML converter user script
« Reply #5 on: August 08, 2014, 06:00:55 AM »
DEPRECIATED (see post below)
Downloader for http://ncode.syosetu.com webnovels:
Sorry, please read the forum rules to see why you can't view spoilers and why you can't post in this forum section. Thank you!

What does it do? You go to the main page for the book you want to download (but should work on chapter pages too), click on the [TXT] link at the top of the page and wait for all chapters to be processed (there is progress info on the title bar and when it's done there is only title of the novel in the title bar). This basically dumps all the chapters into your current page and strips most formatting (no css just plain old divs paraghraphs and headers). You save your novel by telling your browser to save the current page. As I said this version strips most formatting and is intended for converting to epub and other ebook formats.

I plan to make one more version of this script that adds some basic formatting to let you read it as is in the browser... and since I made all kinds of extras (cookie saved memo, furigana that doesn't interfere with Rikaichan and kanji stats) while working on Kurokano I might add them too.

Offline huntermad

  • The Blood Knight!!!
  • J-Regular
  • ***
  • Posts: 219
  • Renommée: 7
  • Gender: Male
  • For fun and profit!!!
    • View Profile
Re: Baka-Tsuki to plain HTML converter user script
« Reply #4 on: August 07, 2014, 04:50:09 AM »
Good job.s
I mostly read BT novele these days so this is perfect!!!

Offline Miguelostos

  • Global Moderator
  • Silver Dragon
  • *****
  • Posts: 7.998
  • Renommée: 5259
  • Gender: Male
    • View Profile
Re: Baka-Tsuki to plain HTML converter user script
« Reply #3 on: August 06, 2014, 05:00:49 PM »
I think that technically, this isn't the right section for what you've posted. I'd think that the right section is the Software board, or the Webhosting board. But seeing as this is rather more specific to the Novel Shoppu, (and because no one really checks the aforementioned boards), for now I'll leave it here.
In fact, I'll sticky it for now, and when, if you eventually do get that web novel script done, post it in this thread, and then change the topic title to something like, 'Novel Rip Scripts'.