Fetch, Parse, and Create Documents for Statically Hosted Files#4398
Merged
timothycarambat merged 16 commits intomasterfrom Oct 1, 2025
Merged
Fetch, Parse, and Create Documents for Statically Hosted Files#4398timothycarambat merged 16 commits intomasterfrom
timothycarambat merged 16 commits intomasterfrom
Conversation
…oad and parse statically hosted files
…ponse content type, as a result unlocked all supported files
…ndard MS Word files in constants.js
timothycarambat
previously requested changes
Sep 19, 2025
| @@ -0,0 +1,31 @@ | |||
| const { WATCH_DIRECTORY } = require("../../utils/constants"); | |||
Member
There was a problem hiding this comment.
Might make more sense to have this single util that downloads files moved to collector/utils/files/index.js since it has to do with files only and we can probably re-use it too
…d into getPageContent | Return explicit argument of captureAs into scrapeGenericUrl in processLink fn
…obal module | Add URL valuidation to downloadURIToFile
remove unused imports
This was referenced Oct 1, 2025
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Pull Request Type
Relevant Issues
resolves #2110
What is in this change?
This PR introduces an enhancement to the document upload modal: the web scraping input field can now fetch, parse, and convert statically hosted files (e.g., https://example.com/a-file.pdf) and API endpoints returning JSON (e.g., https://jsonplaceholder.typicode.com/posts) into application documents.
Currently Supporting:
Currently Supporting:
Text
.txt,.md,.org,.adoc,.rst.html.csv.jsonDocuments
.docx(NOTE:.docfiles are currently not supported, only.docx).odt.pdf.epubPresentations
.pptx.odpSpreadsheets
.xlsxEmail
.mboxAudio
.wav.mp3Video
.mp4.mpegImages
.png.jpgAdditional Information
The core functionality hinges on the Content-Type header of the response. If the mimetype is supported, your file will be pulled and processed.
Developer Validations
yarn lintfrom the root of the repo & committed changes