Location via proxy:   [ UP ]  
[Report a bug]   [Manage cookies]                

Changeset 237376 in webkit


Ignore:
Timestamp:
Oct 24, 2018, 2:05:05 AM (6 years ago)
Author:
jer.noble@apple.com
Message:

TextTrack cues should be updated more often than every 250ms.
​https://bugs.webkit.org/show_bug.cgi?id=190827

Reviewed by Eric Carlson.

Source/WebCore:

Test: media/track/track-cue-timing.html

TextTracks cues are recalculated on the playback timer, which fires at least every 250ms.
In addition to this timer, add a method to MediaPlayer to provide a task which will be
performed at a particular media time, and use this new method to request cues be updated
at the next interesting media time. The next interesting time would be either when the
soonest current cue will end, or when the next non-current cue will start, whichever is
earlier.

(Determining the "next non-current cue" requires new API on PODIntervalTree, as that class
does not have iterators per-se.)

  • html/HTMLMediaElement.cpp:

(WebCore::compareCueIntervalEndTime):
(WebCore::HTMLMediaElement::updateActiveTextTrackCues):

  • platform/PODIntervalTree.h:
  • platform/graphics/MediaPlayer.cpp:

(WebCore::MediaPlayer::performTaskAtMediaTime):

  • platform/graphics/MediaPlayer.h:
  • platform/graphics/MediaPlayerPrivate.h:

(WebCore::MediaPlayerPrivateInterface::performTaskAtMediaTime):

  • platform/graphics/avfoundation/objc/MediaPlayerPrivateAVFoundationObjC.h:
  • platform/graphics/avfoundation/objc/MediaPlayerPrivateAVFoundationObjC.mm:

(WebCore::MediaPlayerPrivateAVFoundationObjC::performTaskAtMediaTime):

  • platform/graphics/avfoundation/objc/MediaPlayerPrivateMediaSourceAVFObjC.h:
  • platform/graphics/avfoundation/objc/MediaPlayerPrivateMediaSourceAVFObjC.mm:

(WebCore::MediaPlayerPrivateMediaSourceAVFObjC::performTaskAtMediaTime):

LayoutTests:

Add a LayoutTest which plays back 6s of captions, each 50 ms in duration, and uses
when the "enter" and "exit" were fired (in media time) to check whether they were missed
or not. The test succeeds if fewer than 50 of the 120 cues were missed.

  • media/track/track-cue-missing-expected.txt: Added.
  • media/track/track-cue-missing.html: Added.
Location:
trunk
Files:
2 added
11 edited

Legend:

Unmodified
Added
Removed

  • trunk/LayoutTests/ChangeLog

    r237367 r237376  
     12018-10-23  Jer Noble  <jer.noble@apple.com>
     2
     3        TextTrack cues should be updated more often than every 250ms.
     4        https://bugs.webkit.org/show_bug.cgi?id=190827
     5
     6        Reviewed by Eric Carlson.
     7
     8        Add a LayoutTest which plays back 6s of captions, each 50 ms in duration, and uses
     9        when the "enter" and "exit" were fired (in media time) to check whether they were missed
     10        or not. The test succeeds if fewer than 50 of the 120 cues were missed.
     11
     12        * media/track/track-cue-missing-expected.txt: Added.
     13        * media/track/track-cue-missing.html: Added.
     14
    1152018-10-23  Ryan Haddad  <ryanhaddad@apple.com>
    216

  • trunk/Source/WebCore/ChangeLog

    r237375 r237376  
     12018-10-23  Jer Noble  <jer.noble@apple.com>
     2
     3        TextTrack cues should be updated more often than every 250ms.
     4        https://bugs.webkit.org/show_bug.cgi?id=190827
     5
     6        Reviewed by Eric Carlson.
     7
     8        Test: media/track/track-cue-timing.html
     9
     10        TextTracks cues are recalculated on the playback timer, which fires at least every 250ms.
     11        In addition to this timer, add a method to MediaPlayer to provide a task which will be
     12        performed at a particular media time, and use this new method to request cues be updated
     13        at the next interesting media time. The next interesting time would be either when the
     14        soonest current cue will end, or when the next non-current cue will start, whichever is
     15        earlier.
     16
     17        (Determining the "next non-current cue" requires new API on PODIntervalTree, as that class
     18        does not have iterators per-se.)
     19
     20        * html/HTMLMediaElement.cpp:
     21        (WebCore::compareCueIntervalEndTime):
     22        (WebCore::HTMLMediaElement::updateActiveTextTrackCues):
     23        * platform/PODIntervalTree.h:
     24        * platform/graphics/MediaPlayer.cpp:
     25        (WebCore::MediaPlayer::performTaskAtMediaTime):
     26        * platform/graphics/MediaPlayer.h:
     27        * platform/graphics/MediaPlayerPrivate.h:
     28        (WebCore::MediaPlayerPrivateInterface::performTaskAtMediaTime):
     29        * platform/graphics/avfoundation/objc/MediaPlayerPrivateAVFoundationObjC.h:
     30        * platform/graphics/avfoundation/objc/MediaPlayerPrivateAVFoundationObjC.mm:
     31        (WebCore::MediaPlayerPrivateAVFoundationObjC::performTaskAtMediaTime):
     32        * platform/graphics/avfoundation/objc/MediaPlayerPrivateMediaSourceAVFObjC.h:
     33        * platform/graphics/avfoundation/objc/MediaPlayerPrivateMediaSourceAVFObjC.mm:
     34        (WebCore::MediaPlayerPrivateMediaSourceAVFObjC::performTaskAtMediaTime):
     35
    1362018-10-23  Fujii Hironori  <Hironori.Fujii@sony.com>
    237

  • trunk/Source/WebCore/html/HTMLMediaElement.cpp

    r237266 r237376  
    16581658}
    16591659
     1660static bool compareCueIntervalEndTime(const CueInterval& one, const CueInterval& two)
     1661{
     1662    return one.data()->endMediaTime() > two.data()->endMediaTime();
     1663}
     1664
    16601665void HTMLMediaElement::updateActiveTextTrackCues(const MediaTime& movieTime)
    16611666{
    … …  
    16771682    // The user agent must synchronously unset [the text track cue active] flag
    16781683    // whenever ... the media element's readyState is changed back to HAVE_NOTHING.
     1684    auto movieTimeInterval = m_cueTree.createInterval(movieTime, movieTime);
    16791685    if (m_readyState != HAVE_NOTHING && m_player) {
    1680         currentCues = m_cueTree.allOverlaps(m_cueTree.createInterval(movieTime, movieTime));
     1686        currentCues = m_cueTree.allOverlaps(movieTimeInterval);
    16811687        if (currentCues.size() > 1)
    16821688            std::sort(currentCues.begin(), currentCues.end(), &compareCueInterval);
    … …  
    17451751        if (!cue->isActive())
    17461752            activeSetChanged = true;
     1753    }
     1754
     1755    MediaTime nextInterestingTime = MediaTime::invalidTime();
     1756    if (auto nearestEndingCue = std::min_element(currentCues.begin(), currentCues.end(), compareCueIntervalEndTime))
     1757        nextInterestingTime = nearestEndingCue->data()->endMediaTime();
     1758
     1759    std::optional<CueInterval> nextCue = m_cueTree.nextIntervalAfter(movieTimeInterval);
     1760    if (nextCue)
     1761        nextInterestingTime = std::min(nextInterestingTime, nextCue->low());
     1762
     1763    if (nextInterestingTime.isValid() && m_player) {
     1764        m_player->performTaskAtMediaTime([weakThis = makeWeakPtr(this), nextInterestingTime] {
     1765            if (weakThis)
     1766                weakThis->updateActiveTextTrackCues(weakThis->currentMediaTime());
     1767        }, nextInterestingTime);
    17471768    }
    17481769

  • trunk/Source/WebCore/platform/PODIntervalTree.h

    r220621 r237376  
    3131#include <wtf/Assertions.h>
    3232#include <wtf/Noncopyable.h>
     33#include <wtf/Optional.h>
    3334#include <wtf/Vector.h>
    3435#include <wtf/text/ValueToString.h>
    … …  
    116117    }
    117118
     119    std::optional<IntervalType> nextIntervalAfter(const IntervalType& interval)
     120    {
     121        auto next = smallestNodeGreaterThanFrom(interval, this->root());
     122        if (!next)
     123            return std::nullopt;
     124
     125        return next->data();
     126    }
     127
    118128    bool checkInvariants() const override
    119129    {
    … …  
    165175            searchForOverlapsFrom<AdapterType>(node->right(), adapter);
    166176    }
     177
     178    IntervalNode* smallestNodeGreaterThanFrom(const IntervalType& interval, IntervalNode* node) const
     179    {
     180        if (!node)
     181            return nullptr;
     182
     183        if (!(interval.high() < node->data().low()))
     184            return smallestNodeGreaterThanFrom(interval, node->right());
     185
     186        if (auto left = smallestNodeGreaterThanFrom(interval, node->right()))
     187            return left;
     188
     189        return node;
     190}
    167191
    168192    bool updateNode(IntervalNode* node) override

  • trunk/Source/WebCore/platform/graphics/MediaPlayer.cpp

    r237266 r237376  
    15381538#endif
    15391539
     1540bool MediaPlayer::performTaskAtMediaTime(WTF::Function<void()>&& task, MediaTime time)
     1541{
     1542    return m_private->performTaskAtMediaTime(WTFMove(task), time);
     1543}
     1544
    15401545#if !RELEASE_LOG_DISABLED
    15411546const Logger& MediaPlayer::mediaPlayerLogger()

  • trunk/Source/WebCore/platform/graphics/MediaPlayer.h

    r237266 r237376  
    572572#endif
    573573
     574    bool performTaskAtMediaTime(WTF::Function<void()>&&, MediaTime);
     575
    574576private:
    575577    MediaPlayer(MediaPlayerClient&);

  • trunk/Source/WebCore/platform/graphics/MediaPlayerPrivate.h

    r237266 r237376  
    282282    virtual AVPlayer *objCAVFoundationAVPlayer() const { return nullptr; }
    283283#endif
     284
     285    virtual bool performTaskAtMediaTime(WTF::Function<void()>&&, MediaTime) { return false; }
    284286};
    285287

  • trunk/Source/WebCore/platform/graphics/avfoundation/objc/MediaPlayerPrivateAVFoundationObjC.h

    r237364 r237376  
    331331    AVPlayer *objCAVFoundationAVPlayer() const final { return m_avPlayer.get(); }
    332332
     333    bool performTaskAtMediaTime(WTF::Function<void()>&&, MediaTime) final;
     334
    333335    RetainPtr<AVURLAsset> m_avAsset;
    334336    RetainPtr<AVPlayer> m_avPlayer;

  • trunk/Source/WebCore/platform/graphics/avfoundation/objc/MediaPlayerPrivateAVFoundationObjC.mm

    r237364 r237376  
    32893289}
    32903290
     3291bool MediaPlayerPrivateAVFoundationObjC::performTaskAtMediaTime(WTF::Function<void()>&& task, MediaTime time)
     3292{
     3293    if (!m_avPlayer)
     3294        return false;
     3295
     3296    __block WTF::Function<void()> taskIn = WTFMove(task);
     3297
     3298    [m_avPlayer addBoundaryTimeObserverForTimes:@[[NSValue valueWithCMTime:toCMTime(time)]] queue:dispatch_get_main_queue() usingBlock:^{
     3299        taskIn();
     3300    }];
     3301    return true;
     3302}
     3303
    32913304NSArray* assetMetadataKeyNames()
    32923305{

  • trunk/Source/WebCore/platform/graphics/avfoundation/objc/MediaPlayerPrivateMediaSourceAVFObjC.h

    r236572 r237376  
    235235#endif
    236236
     237    bool performTaskAtMediaTime(WTF::Function<void()>&&, MediaTime) final;
     238
    237239    void ensureLayer();
    238240    void destroyLayer();
    … …  
    273275    RetainPtr<id> m_timeJumpedObserver;
    274276    RetainPtr<id> m_durationObserver;
     277    RetainPtr<id> m_performTaskObserver;
    275278    RetainPtr<AVStreamSession> m_streamSession;
    276279    RetainPtr<CVPixelBufferRef> m_lastPixelBuffer;

  • trunk/Source/WebCore/platform/graphics/avfoundation/objc/MediaPlayerPrivateMediaSourceAVFObjC.mm

    r236875 r237376  
    11111111#endif
    11121112
     1113bool MediaPlayerPrivateMediaSourceAVFObjC::performTaskAtMediaTime(WTF::Function<void()>&& task, MediaTime time)
     1114{
     1115    __block WTF::Function<void()> taskIn = WTFMove(task);
     1116
     1117    [m_synchronizer addBoundaryTimeObserverForTimes:@[[NSValue valueWithCMTime:toCMTime(time)]] queue:dispatch_get_main_queue() usingBlock:^{
     1118        taskIn();
     1119    }];
     1120    return true;
     1121}
     1122
    11131123}
    11141124
Note: See TracChangeset for help on using the changeset viewer.