Unveiling the unconventional: 5 lesser-known obstacles in Flutter development

This article has been translated from its original publication at https://habr.com/ru/companies/surfstudio/articles/730340/

Our streaming solution was built on the Flutter platform (there is it - The Hole project). Along the way, we encountered a number of challenges, including limitations in video player packages, difficulties with picture-in-picture implementation, and platform-specific errors.

Our relative inexperience with Flutter, coupled with the technology's young age (we started the project in 2020), led to some specific bugs and hurdles. Additionally, we encountered certain obstacles that were not Flutter-specific but rather native-specific, so they're relevant to native devs too.

Let's dive into the details!

Introduction: Streaming on Flutter? Ugh...

Well, it's time for me to introduce myself. I'm Timur Tkharkakhov, a Flutter developer, and today we're going to tackle the challenges that Dash so diligently throws at those who venture into streaming development.

Let me share a bit of backstory. I recently joined Surf as a developer and discovered that my upcoming project would involve video streaming. I'll be honest, I was skeptical. My experience as an Android developer had taught me that working with video can be a challenging and treacherous endeavor.

And here I was, faced with Flutter, a framework known for its own intricacies. Not to mention the client's insistence on having background playback and picture-in-picture features. My initial fears were partially confirmed when I inherited a bug from my colleagues: the app crashed into a black screen right after installation, on its very first run.

However, as I delved deeper into the project, my perspective on Flutter underwent a significant transformation: things turned out to be not as dire as I had anticipated. 

A Brief Overview of Flutter's Interaction with Native Platforms

For those familiar with Flutter, feel free to skip ahead to the next section.

Flutter is primarily focused on providing a robust UI framework. Its core objective is rendering UI components, which leads to certain implementation peculiarities when it comes to communicating with the outside world. In our case, the "outside world" refers to Android and iOS, although Flutter also supports Web and Desktop platforms.

Under the hood, Flutter utilizes the Dart virtual machine. Code executed within this virtual machine operates in an isolated environment, allowing for the utilization of not only common UI elements but also the business logic of the application across different platforms.

However, due to this isolated environment, direct access to many "native" features from Dart is not available. To bridge this gap, the Flutter team has introduced a mechanism called channels for interaction with the native platform. With a bit of documentation study, one can establish communication between Flutter and the platform, enabling the invocation of native code from Flutter and Dart code from the native side.

Now, let's delve into a particularly significant feature for our narrative: the ability to integrate native UI components (such as Views and UIViews) into Flutter. These components can be seamlessly embedded within the widget tree and manipulated through Dart code. This functionality allows developers to leverage existing community-developed solutions, reducing the need to build complex components from scratch. A prime example of this is the google_maps plugin. Likewise, one of the main protagonists of this article, video_player, operates in a similar manner, utilizing the well-known native components AVPlayer for iOS and ExoPlayer for Android under the hood.

Obstacle #1: video_player Package 

It was this plugin that the Surf team chose at the beginning of project development. I have to say that so far, more than two years later, there are almost no serious and high-quality alternatives to it. Only flutter_vlc seems to be potentially interesting.

Challenges with the video_player

The video_player package, while providing basic functionality, lacks certain essential features. For instance, it does not offer the ability to collect detailed playback status logs, limiting the visibility into the inner workings of the player.

One notable issue revolves around the divergent behavior of the player when reconnecting to the internet on Android and iOS. On Android, the only way to resume video playback after a reconnection is to recreate the player instance. On the other hand, iOS lacks an error signal indicating a lack of internet connectivity, necessitating reliance on third-party package data.

Moreover, when faced with an unstable connection, inconsistencies arise with buffering and loading flags, and the absence of comprehensive error messages adds to the troubleshooting challenges.

The surge in the user base has resulted in an increased number of bugs in the crashlist—a situation previously unseen. The root cause of many of these problems can be traced back to issues within the video player plugin.

Our Experience with the video_player Package in The Hole Project

Our journey with the video_player package on The Hole project was met with both challenges and opportunities for improvement. Initially, we raised an issue in the package repository, but it became evident that not all feature requests and bug fixes would receive immediate attention from the support team.

As a result, our team decided to take matters into our own hands. We forked the plugin, conducted thorough research, and compiled a list of crucial features that required refinement. Some of these included:

• Enhanced error messages to provide more informative feedback.
• The ability to initiate playback from a specific position.
• Control over buffer size and buffer return.
• Comprehensive collection of playback statistics, such as channel width, buffering time, and lost frames.
• Potential support for playlists and other related features.

While we didn't implement all of these features, the ones we did implement were not without their imperfections. Nonetheless, these enhancements allowed us to address most of the issues faced by our users.

In retrospect, we have no regrets about using the video_player package. It facilitated a quick start to the project and allowed us to validate our hypotheses. However, it's important to note that significant time and effort were invested in reworking the package. Consequently, depending on the team's expertise with the target platforms, the idea of developing a custom plugin from scratch may be worth considering.

Obstacle #2: Background Playback Quandary

Or "Easy to Learn, Hard to Master"

Implementing background playback functionality initially seemed straightforward using the audio_service library. However, as we delved deeper into the development process, we encountered a series of nerve-wracking bugs tied to various other features.

One notable bug that left a lasting impression was the occurrence of a "black screen" upon the first launch of the application after installation from the stock on specific Samsung and OnePlus devices. Despite our efforts to communicate with the maintainer and the emergence of similar behavior reported by other developers, the bug never made its way into the upstream codebase.

Communication with instructions on how to reproduce the bug

The issue stemmed from a misconnection between the application and the FlutterEngine during the startup of the Activity, particularly when the Activity had been previously started within a Service where events like headset button presses were handled. Drawing upon our native expertise and understanding of Android component lifecycles, we were able to pinpoint this problem.

Consequently, we established another valuable addition to our fork in the pubspec on GitHub. Subsequently, we further refined the library's functionality to ensure proper interaction with headsets, as well as handling the muting and restoration of sound during incoming calls.

Obstacle #3: Picture-in-Picture Predicament

A significant portion of the code required for the aforementioned enhancements was written in native languages:

• Android: Java and Kotlin
• iOS: Swift and Objective-C

Yes, you've read it right—Objective-C! It may be a slight exaggeration (though not by much), but it seems that the iOS portion of the packages supported by Google teams in the flutter/packages repository predominantly utilize Objective-C. Finding an iOS developer willing and enthusiastic about fixing or writing Objective-C code can be a genuine challenge. However, as a Flutter developer at Surf, it's all part of our T-shaped development approach.

In this particular domain, one of our team members took the plunge (with the guidance of an iOS developer, of course, as we value mentorship). Their mission was to write the iOS segment of the plugin dedicated to supporting the picture-in-picture feature. Despite the code being in Swift, we had to delve deep into the inner workings of the video_player package, predominantly written in Objective-C, to establish a harmonious connection.

Our objectives were as follows:

• Retrieve the textureId utilized by the player plugin to render video frames.
• Pass this textureId to our plugin.
• Utilize the textureId to establish a linkage with the player.
• Throughout this process, we navigated the intricacies of the iOS app lifecycle, Flutter widgets, and background playback considerations.

It was undoubtedly an ambitious undertaking.

On a related note, with the release of Flutter 3.7, the developers announced the migration of these packages to Swift.

Obstacle #4: System Resource Management

When developing an application centered around video catalog playback, having a vertical video catalog with previews becomes essential. In Flutter, achieving this is a breeze, eliminating the need to tangle with RecyclerView on Android or UICollectionView and its counterparts on iOS. Flutter offers a versatile ListView widget capable of lazily displaying a conditionally infinite list.

Suppose you desire each item in the list to have a video controller and commence playback upon gaining focus. Initially, our feed's previews instantly began playing upon receiving focus. However, it's crucial to bear in mind that Flutter's "everything is a widget" concept, while streamlining the design, layout, and reusability of UI elements, necessitates careful attention in exceptional scenarios.

Widgets in Flutter possess their own lifecycle, extending beyond the boundaries of application screens due to the "everything is a widget" principle. Unlike Fragments, Activities, or ViewControllers, Flutter lacks direct equivalents. Consequently, specific behaviors at the intersection of widgets and native components become feasible.

How many "live" instances of the video player do you think we can accommodate within the ListView example? The answer is not as straightforward as it may initially appear. To find a solution, we embarked on a quest for answers, scouring Stack Overflow, package sources, and even (sic!) documentation, while conducting extensive research online.

Our conclusion to the question is that it is optimal to support only one active player instance in the application. This answer holds true universally, not just for Flutter but also for native apps. You may have noticed that the YouTube app experiences a noticeable delay when starting the next preview as you scroll through the feed. This delay is intentional as YouTube prioritizes stability over immediate playback.

When there are multiple active player instances present, such as on the feed, an open episode screen, and potentially a third instance, it can exhaust available resources, resulting in video malfunction or failure to work altogether.

Lesson Learned

It is crucial to manage device resources wisely and release them when they are not in use. It's important to remember that not all users have high-end devices with ample memory and performance capabilities. Taking into consideration the limitations of different devices can greatly enhance the user experience.

Furthermore, it's essential to remember that our app is not the only one installed on users' devices. By optimizing resource usage, we not only improve our own app's performance but also contribute to the overall smooth functioning of the device. Users will appreciate the effort put into optimizing the app and providing a seamless experience.

Therefore, it is valuable to provide feedback not only when things go wrong but also when things work well. Recognizing and acknowledging positive experiences can motivate developers and contribute to the improvement of future app versions.

Obstacle #4: Peculiarities Of Different Platforms

It's important to consider that various devices utilize different hardware decoders. While a video may play smoothly on the majority of devices, it may encounter issues or perform sluggishly on a small number of them.

Moreover, not all device drivers accurately report their capabilities. For instance, we've come across budget Android devices that falsely claim to support 1080p video playback, but in reality, they struggle to handle it. This realization prompted us to incorporate a lost-frame collection feature into the player plugin.

iOS also presents its own set of peculiarities. In the case of AVPlayer, we can only specify the desired buffer size for video, but we cannot explicitly define which one will be utilized. The behavior of AVPlayer's buffer size selection seems to vary, adding an element of uncertainty.

Another noteworthy example relates to HLS playlists, which indicate the available video quality options. While higher quality options are automatically accessible, we wanted to empower users to manually select a specific quality, especially when it can help conserve data while roaming. Although AVPlayer offers a method for setting the preferred quality, it operates based on internal algorithms that determine what it deems optimal in practice.

Lesson Learned

Remember the platform-specific peculiarities. Some functionalities may be available only on one platform and not on others. If you encounter unexpected behavior or issues with a particular method on a specific platform, it's important to refer to the documentation or examine the package's source code. Thankfully, many packages in the development community are open source, providing you with the opportunity to explore their implementation. 

Key Takeaways from Streaming on Flutter:

• YouTube doesn't charge you for the background playback feature without reason .
• While Flutter offers convenience in UI and animations, it's crucial to have a solid foundation in native development for a comprehensive understanding.
• If you begin your mobile development journey with Flutter, acquiring knowledge about platform-specific peculiarities will give you a competitive edge.

In Place of a Conclusion

There are numerous complex, buggy, and fascinating functionalities we explored, some of which didn't make it to the final release:

• Protection against screenshots and screen recording
• Watermark overlay on recorded videos
• Animation during screen rotation on iOS
• Challenges with FFmpeg while video clipping
• User reactions while watching videos
• Exciting UI components and animations
• AirPlay and Chromecast support
• Accessing videos from different parts of the world, even where it shouldn't be accessible
• Prerolls for commercials (advertising drives commerce)
• Migration to Flutter 2 and null safety (a painful process)
• Upgrading to Flutter 3 (a smoother experience)
• Transitioning to Elementary (without any issues)
• Extensive back-end optimizations, redesigning, testing, and debugging

But we'll save those stories for another time.