Work in progress, please help improve this page

Introduction

This is our Sovereign Tech Fund Application creation page.

Information for Developers

Please familiarize yourself with [STF](​https://www.sovereigntechfund.de/programs/applications). Then look over the existing projects below, and either add yourself to one or add your own project idea. Do not hesitate to fix this page if you see any issues.

Potential projects (brainstorm)

Old Bug Fixing

Description: Over the last decade bugs have accumulated that noone is fixing. This entry is to get these fixed.

[Trac search showing the current list](​https://trac.ffmpeg.org/query?status=new&status=open&status=reopened&type=defect&reproduced=1&time=Jan+1%2C+2010..Jan+1%2C2023&max=1000&order=priority)

Expected results: Reduce the number of long standing open bugs upto 2022 which have been reproduced before 2024-02-03.

Duration: 8 months

Payment: 41390€ if all are fixed. More precisely the payment will depend on the time since the bug was created:

• 2011 7 bugs 190€ per bug fixed
• 2012 36 bugs 180€ per bug fixed
• 2013 28 bugs 170€ per bug fixed
• 2014 29 bugs 160€ per bug fixed
• 2015 21 bugs 150€ per bug fixed
• 2016 52 bugs 140€ per bug fixed
• 2017 31 bugs 130€ per bug fixed
• 2018 16 bugs 120€ per bug fixed
• 2019 18 bugs 110€ per bug fixed
• 2020 16 bugs 100€ per bug fixed
• 2021 22 bugs 90€ per bug fixed
• 2022 28 bugs 80€ per bug fixed

Left over money after 8 months, shall be rolled over into the next "Old Bug Fixing" Project

Developer: anyone who likes to help and has signed the paperwork

Move to Gitlab

Description: Gitlab has a lot of benefits including automatic CI, easy merges and in built issue tracking. Gitlab/Github development is a way a lot of Open Source developers operate in 2024, as mailing lists are difficult to use (especially with recent changes in Gmail and other major providers). This will help the project by getting patches merged faster as they are tested automatically. dav1d has made good use of Gitlab.

Expected results: trac and mailing list development gone

Duration: whatever time it takes

Payment: Ask for bids from actual sysadmins

Developer: ?

Remove MMX

Description: MMX code causes lots of issues with emms and is not relevant in 2024

Expected results: MMX Gone

Duration: whatever time it takes

Payment: €20k

Developer:Kieran Kunhya' ?

Coverity Issues

Description: Coverity is a static code analysis system that is used to analyze FFmpeg code to find bugs with an emphasis on quality and security issues. There are currently 673 outstanding issues identified by Coverity. Some of these issues are false positives while others could open the door to security vulnerabilities.

Milestones: There will be 3 pairs of milestones, the first of each pair will be about Fixing the outstanding real issues, improve code where simple shortcommings have been noticed and patches will be submitted. The 2nd of each pair will be about merging the patches to git master.

Expected results: all issues become classified, most real issues become fixed, tangentially related things found in the work will also be fixed. It is understood that some issues may be too difficult to classify or fix and that a small number of patches may be rejected and not merged for reasons outside the scope of this work.

Duration: 12 Months

Payment: 50,000 EUR

Developer:

Fix ossfuzz coverage

Description: ossfuzz coverage testing is currently broken. Even though this hasn't been fixed for some time, it'd still be very important to know what areas are covered by the fuzzers. A big problem was that the ossfuzz infra had many restrictions from timeouts, very slow build within docker, and limitations on everything with shared libs not allowed and the requirement of a single binary per test. It may be that some of these restrictions are no longer applicable. Fixing this can be done within the ossfuzz infra or also on a newly setup system that is free of restrictions.

Expected results: working, automated and robust ossfuzz coverage testing

Duration: ?

Payment: Choosen by Developer when she adds herself

Developer: NA

Merge FFmpeg Forks

Description: FFmpeg has been forked by paul, it is in the best interest of our users that all improvement and bug fixes are integrated in main FFmpeg.

Expected results: bring all changes from a specified fork through the review process on [ffmpeg devel](https://ffmpeg.org/mailman/listinfo/ffmpeg-devel/). Each change may be accepted or rejected with explanation. Alternatively, we could instead specify here the number of hours of work and leave it open how much would be merged

Duration: 6 months

Payment: 20 € per commit, limited to a maximum of 10,000€ (If more commits happen this limit should be higher)

Developer:

Cleanup HEVC Decoder

Description: Current HEVC decoder works for simple cases but missing some key features, and hard to extend. For example, it doesn't have tile level parallel, and frame level parallel doesn't work with multi-tile streams. It missing multi-layer extension support and hard to add in current state. It needs some cleanup and refactor. And since we got a native VVC decoder, they can share more on the design and optimization.

Expected results: Have a clean structure, make it scalable (easy to add multi-layer extension but don't need to implement that for this project), and improve the performance.

Duration: 12 months

Payment:

Developer: Zhao Zhili

Improving libswscale

Description: Work on improving the libswscale API, internal refactoring, and fix support for currently broken cases

Expected results: Spend 6 months working fulltime on improving all aspects of libswscale - refactoring and modernizing the API, splitting off and disentangling internal components, wrapping them inside nicer internal abstractions, and ultimately refactoring the architecture to allow us to support things like HDR or wide gamut color spaces. Also fixing current known bugs and limitations, removing self-modifying code, etc.

Duration: 6 Months

Payment: 60,000 EUR

Developer: Niklas Haas

Maintain FFv1

Description: FFv1 has become one of the entirely open and free de facto standard lossless codecs for video archival. There has however been insufficient resources to maintain it, Issues, Ideas, Bugs, ... are missing the resources needed to take care of them.

Expected results: Fuzzing, Bugfixing, and support for floating point images. The later requires investigation of different ways to compress floating point values.

Duration: 12 Months

Payment: 25,000 EUR (maintaining FFv1 is of course worth funding, but who agreed to this amount 1 hr before submission?)

Developer: Michael Niedermayer

Separate libpostproc

Description: Several developers want libpostproc to be maintained outside the main FFmpeg repository. Over many years noone has volunteered to do this.

Expected results: Split libpostproc out into its own git repository which is maintained independent of the FFmpeg community. Have functioning build and selftest systems. The bug tracking system may remain shared to avoid having it become controlled by github.

Duration: 8 Months

Payment: 15,000 EUR

Developer: Michael Niedermayer

STF Application

# FFmpeg Project Description

1) Describe the technology this effort will support.

The proposed effort aims to support FFmpeg, a leading multimedia framework recognized for its versatile capabilities.

FFmpeg serves as a comprehensive solution for decoding, encoding, transcoding, multiplexing, demultiplexing, streaming, filtering, and playback of a wide array of multimedia formats. It stands out by seamlessly handling formats from ancient, obscure standards to cutting-edge ones, making it a cornerstone in the multimedia processing landscape.

2) Where is this technology being used? Why is it relevant and critical? How does this technology serve the public interest?

FFmpeg is a ubiquitous technology, deeply embedded in the fabric of modern multimedia processing and touching the lives of millions every day. FFmpeg is an integral component in a vast array of end-user devices, including desktops, laptops, mobile phones, smartwatches, and smart TVs. FFmpeg also powers major video distribution and streaming platforms like YouTube, Vimeo, AWS, and Microsoft Azure, shaping the landscape of online content consumption.

The technology extends its influence to TV broadcasts over IP, enabling seamless delivery of multimedia content to a global audience. Beyond mainstream applications, FFmpeg is employed in diverse sectors, from archival projects to the Mars Perseverance Rover, showcasing its adaptability and reliability.

The relevance and criticality of FFmpeg lies in its role as the backbone of multimedia processing. It serves the public interest by facilitating the seamless functioning of everyday technologies, from video playback in web browsers like Firefox and Chromium, to applications like Kodi, MPlayer, and OBS Studio. Furthermore, FFmpeg plays a crucial role in public sector use cases, website interactions, and even in the transmission of multimedia content within emails. In essence, FFmpeg's impact is far-reaching, touching both the private and public spheres. FFmpeg's reliability is paramount for the secure and efficient functioning of digital multimedia.

3) What are the problems the proposed work is trying to address? What activities do you propose and how do they address the problems described? How do those activities fit into [STF’s mission](​https://www.sovereigntechfund.de/mission)?

The proposed work addresses critical challenges in maintaining FFmpeg's sustainability, security, and innovation. Activities include:

• Security Fixes and Hardening: Addressing bugs and enhancing security through improvements of the fuzzing system.
• Administration of FFmpeg Infrastructure: Ensuring robust infrastructure management for reliability and performance.
• Improvements in Codecs, Formats, and Filters: Enhancing existing implementations.

These activities align with STF’s mission by supporting the maintenance and security of open source components and the strengthening the Open Source ecosystem.

4) Who is going to be performing the work and how are they qualified to do so?

The work will be performed by key FFmpeg project contributors and other experienced developers. Their qualifications have been demonstrated through years of active involvement in FFmpeg development, participating in diverse tasks including code contributions, code review, user support, and administration.

5) How is the technology maintained or governed? Is there a community behind the technology, and do they approve of the work?

FFmpeg's maintenance and governance in based on a flat hierarchy, with decisions being made transparently through written communications in mailing lists. The FFmpeg project boasts a robust community of administrators, maintainers, and contributors who actively engage in the decision-making processes through consensus-driven discussions. The proposed work aligns with the community's focus on quality, security, and openness and will/has been approved following the community's operating governance processes. (Note that new tasks were added to the wiki 1 hour before submission).

6) Describe the Milestones Using the Template below

Classify and fix outstanding issues identified by Coverity and improve related code.

Overview

Coverity is a static code analysis system that is used to analyze FFmpeg code to find bugs with an emphasis on quality and security issues. There are currently 673 outstanding issues identified by Coverity (​https://scan.coverity.com/projects/ffmpeg?tab=overview). Some of these issues are false positives while others could open the door to security vulnerabilities.

The objective of this work is to identify the Coverity issues that are not false positives, and fix 90% of them, split into three tranches of 30%.

Any issue that require modules rewrite or redesign are outside the scope of this SoW. Any related issues will be fixed / cleaned up if practical. An example of this is the use of literal 4 instead of a named identifier for the number of planes in a false positive "out of array access" issue. The issue was closed as false positive yet patches to make the code more robust and easier to understand where submitted none the less.

Developer: Michael Niedermayer <​michael-ffwork@niedermayer.cc>

Duration: 12 months

Milestones

Milestone C

• Description: Review all outstanding Coverity issues and, for each one, determine whether it is a false positive.

• Deliverables: List of both false positive and potentially real issues posted to the FFMPEG dev mailing list.

Milestone S1

• Description. Fix the first 30% tranche of outstanding real issues.

• Deliverables. Patches submitted for review to the FFMPEG dev mailing list.

Milestone M1

• Description. Commit to the FFMPEG master branch 90% of the changes from S1.

• Deliverables. Link to the commits containing the changes.

Milestone S2

• Description. Fix the second 30% tranche of outstanding real issues.

• Deliverables. Patches submitted for review to the FFMPEG dev mailing list.

Milestone M2

• Description. Commit to the FFMPEG master branch 90% of the changes from S2.

• Deliverables. Link to the commits containing the changes.

Milestone S3

• Description. Fix the third 30% tranche of outstanding real issues.

• Deliverables. Patches submitted for review to the FFMPEG dev mailing list.

Milestone M3

• Description. Commit to the FFMPEG master branch 90% of the changes from S2.

• Deliverables. Link to the commits containing the changes.

Maintain the FFV1 Codec

Overview

FFv1 has become one of the entirely open and free de facto standard lossless codecs for video archival. There has however been insufficient resources to maintain it and there are, as a result, several outstanding issues and pull requests to address:

​https://github.com/FFmpeg/FFV1/issues ​https://github.com/FFmpeg/FFV1/pulls

Developer: Michael Niedermayer <​michael-ffwork@niedermayer.cc>

Duration: 12 months

Milestones

Fuzzing

• Description: Add encoder and decoder fuzzing to find new security issues.

• Deliverables: Scripts and input files necessary for fuzzing

Bug fix

• Description. Fix 90% of outstanding bugs and merge 90% of outstanding pull requests intended to fix outstanding bugs.

• Deliverables. Link to commits on the FFMPEG master branch containing the changes.

Add support for floating point samples

• Description. The objective is to add support for floating point samples

to FFv1. This would allow lossless storage of a wider range of content in archives using FFv1.

• Deliverables. List of git commits on the FFMPEG master branch adding

broadening the support of formats in the specification, encoder and decoder

Improving and modernizing libswscale

Overview

libswscale is an old and integral component of FFmpeg which is relied on, in some form, by almost every user. Despite this, it is one of the hardest parts of FFmpeg to extend and understand, owing to its historical growth in complexity. The aim of this project will be to alleviate some of the issues and start work on bringing libswscale into a cleaned up, more maintainable state.

Developer: Niklas Haas <​ffmpeg@haasn.xyz>

Duration: 12 months

Milestones

Design new high-level API

• Description: Develop and design a new high-level API

(libswscale/avscale.h), which is capable of abstracting away the internal messy, stateful details of libswscale. This API *should* ideally also be written in such a way that we can internally use zscale or libplacebo instead of libswscale, possibly even at the same time (e.g. if incompatible, complex, multi-step transformations are requested), be almost entirely stateless (backed by some sort of "state cache" object?), and allow for implementing things like slice or frame threading at a higher level (thus freeing libswscale from the burden of managing cascaded contexts internally).

• Deliverables: RFC of complete API draft posted to mailing list, with at

least general consensus that this API represents a step in the right direction.

Implement backends for new API

• Description. The previously designed API should be fully implemented,

hooked up to at least libswscale, in particular, in a way that solves the existing issues of libswscale's statefulness (e.g. errors after reconfiguring to different bit depths or color spaces). If time allows, also hook it up to libplacebo and/or zscale as a proof of concept, possibly just to provide an initial backing for operations which libswscale does not yet support (e.g. colorspace conversions, GPU frame support).

• Deliverables. Series merged to git master introducing new API and backing

implementation, vf_scale successfully ported to avscale.h.

Extend and fix regression tests

• Description. Fix existing known issues with libswscale's self-tests,

and extend them by adding coverage for more paths. In particular, add checkasm support for existing assembly primitives. Fix any issues that are uncovered by the additional tests.

• Deliverables. Patch series merged to git master, closing existing

issues (e.g. #10824), adding testing for at least half of assembly routines, and fixing whatever is needed to ensure that the assembly passes checkasm.

Remove legacy code paths

• Description. In the interest of making subsequent refactoring work more

achievable, libswscale should first be slimmed down to the essentials of what's needed for reasonably modern platforms. Older code, assembly routines for rarely used platforms, legacy speed hacks like runtime-generated MMX code etc. should be identified, analyzed to figure out what actual benefit they provide, and proposed for removal.

• Deliverables. A list of proposed removals, plus justification (e.g. in

the form of benchmarks or platform statistics) is posted to the mailing list. All proposals are either accepted by the community (and merged), or rejected (and discarded).

Separate libpostproc from the main FFmpeg repository

Overview

libpostproc has existed in FFmpeg since 2003. FFmpeg and libpostproc have since diverged in their design, and they no longer fit well into the same repository. The objective of this work is to seperate libpostproc from FFmpeg cleanly so they both have independant build and testing systems while ensuring that FFmpeg can still link to libpostproc.

Developer: Michael Niedermayer <​michael-ffwork@niedermayer.cc>

Duration: 12 months

Milestones

Milestone S

• Description: Separate libpostproc from FFmpeg.

• Deliverables: Link to a public git repository that contains libpostproc with history which can be build and linked with FFmpeg.

Milestone T

• Description. Test system which allows testing most features of libpostproc.

• Deliverables. Code merged into new libpostproc repository.

Milestone F

• Description. Remove libpostproc from FFmpeg.

• Deliverables. libpostproc code removed from FFmpeg repository.

Accepted STF Projects

The following projects from the above list have been granted for this year's sponsoring:

• Classify and fix outstanding issues identified by Coverity and improve related code
• Maintain the FFV1 Codec
• Improving and modernizing libswscale
• Separate libpostproc from the main FFmpeg repository