Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Implement volume controlling AudioProcessor, with independent control of channel volumes #2659

Closed
platinum7919 opened this issue Apr 8, 2017 · 50 comments
Closed

Implement volume controlling AudioProcessor, with independent control of channel volumes #2659

platinum7919 opened this issue Apr 8, 2017 · 50 comments
Assignees
@andrewlewis
Labels
enhancement low priority

Comments

@platinum7919
Copy link

I need a way to switch left and right audio volume separately. Similar to
https://developer.android.com/reference/android/media/MediaPlayer.html#setVolume(float, float)
Currently, I can't find a way to do this in Exoplayer

Using exoplayer2.3.1
Using android 6.0.1+
@ojw28
Copy link
Contributor

ojw28 commented Apr 10, 2017 •
edited
Loading

The stereo variant for setting volume was deprecated in the platform, at least on AudioTrack. See here. I do wonder what the recommended approach is though. @andrewlewis - Any ideas?

@Avetri
Copy link

Avetri commented Apr 10, 2017

@ojw28 Separated volume control for stereo channels is an often requirement of TV operator for its boxes.
Based on the old technique for analogue TV. As example English and Spain tracks were broadcasted in the same audio track in different channels for southern states of USA.

@andrewlewis andrewlewis self-assigned this Apr 10, 2017
@andrewlewis
Copy link
Collaborator

The platform doesn't provide support for this [internal bug 37200638]. We could implement it as an AudioProcessor that allows the application to set a volume multiplier for each channel, but it likely won't happen for a while. Feel free to send a pull request.

We have an existing audio processor called ChannelMappingAudioProcessor, which would address the use case of taking stereo and replacing one channel with the other. It might make sense to add functionality there for setting the volumes of output channels too.

@andrewlewis andrewlewis mentioned this issue Sep 6, 2017
Closed
@xdestiny110 xdestiny110 mentioned this issue Oct 10, 2017
Closed
@andrewlewis andrewlewis mentioned this issue Jan 3, 2018
Closed
@andrewlewis andrewlewis changed the title Control stereo speaker volume (L and R) separately Implement volume controlling AudioProcessor, with independent control of channel volumes Jan 3, 2018
@AquilesCanta AquilesCanta mentioned this issue Feb 7, 2018
Closed
@mdtuyen
Copy link

mdtuyen commented Feb 27, 2018

Hi, I found a solution, You can edit AudioTrack.java file to use setStereoVolume insteal setVolume.
However setStereoVolume have a bug: it not work on some android version.

https://stackoverflow.com/questions/10462364/android-audiotrack-setstereovolume-not-working

@roundsmapp
Copy link

Hi, I do have the very same problem as @platinum7919, i.e., the need to control left and right audio volumes for ExoPlayer.
Everything else going fine, including looping (gapless) mp3 palying.
Using exoplayer 2.7.1 (SimpleExoPlayer class).

Any forecast for such method availability? Or any intermediate solution?

@andrewlewis
Copy link
Collaborator

We don't have any plans to support this at the moment, as it seems quite niche. For the case of different languages in left/right channels, using ChannelMappingAudioProcessor lets you achieve the desired effect. For changing the volumes of channels independently in general, I suggest implementing a custom AudioProcessor that copies each channel and applies the relevant scaling.

@roundsmapp
Copy link

@andrewlewis , thank you for your response. Is there any place where I can see a custom implementation of AudioProcessor with separate channels? I've been looking for documentation and didn't find where I can make those channels work.

@andrewlewis
Copy link
Collaborator

I'd start by reading ChannelMappingAudioProcessor to see how it works, then make a copy of it and modify it to multiply each sample being copied by a factor associated with its channel. The input/output data to/from the processor is interleaved 16-bit PCM, so it might be best to access the input buffer as a ShortBuffer rather than a ByteBuffer.

To use your custom AudioProcessor with SimpleExoPlayer you can override DefaultRenderersFactory.buildAudioProcessors in the renderers factory you pass to the constructor.

@roundsmapp
Copy link

@andrewlewis , I guess I got it. Thank you!

@roundsmapp
Copy link

@andrewlewis , I was able to effectively separate the channels. But the player has a weird behaviour. First of all, if I configure AudioProcessor (ChannelMappingAudioProcessor) with only 2 channels, I can isolate the left one, but not the right one.

If I configure it with 3 or more channels, then I can isolate both, making the audio come out from just one channel or from both.

Independently of that, I have another problem. From the channel that is OFF, there is a strange noise coming out of it, while the other channel is playing the audio well.

Do you see any explanation?

@andrewlewis
Copy link
Collaborator

It's hard to say what's going on without seeing the code. Could you post it somewhere?

@roundsmapp
Copy link

@andrewlewis , Here you have it:

public void queueInput(ByteBuffer inputBuffer) {
int position = inputBuffer.position();
int limit = inputBuffer.limit();
int frameCount = (limit - position) / (2 * channelCount);
int outputSize = frameCount * outputChannels.length * 2;
if (buffer.capacity() < outputSize) {
buffer = ByteBuffer.allocateDirect(outputSize).order(ByteOrder.nativeOrder());
} else {
buffer.clear();
}
int i;
while (position < limit) {
for (int channelIndex : outputChannels) {
i=0;
buffer.putShort(inputBuffer.getShort((position + 2 * i) * channelIndex));
i++;
}
position += channelCount * 2;
}
inputBuffer.position(limit);
buffer.flip();
outputBuffer = buffer;
}

outputChannels has the values 1 or 0 for the channels.

@andrewlewis
Copy link
Collaborator

The index calculation (position + 2 * i) * channelIndex) looks wrong, as it doesn't make much sense to multiply the position in the buffer by the channel index.

Could try taking ChannelMappingAudioProcessor verbatim and replacing the copying loop with the following (warning: untested)?

while (position < limit) {
  for (int i = 0; i < outputChannels.length; i++) {
    short sample = inputBuffer.getShort(position + 2 * outputChannels[i]);
    sample = (short) (sample * outputChannelVolumes[i]);
    buffer.putShort(sample);
  }
  position += channelCount * 2;
}

Some more care probably needs to be taken with scaling the sample but hopefully this is enough to get started. ChannelMappingAudioProcessor also has logic to make it inactive if channels aren't modified, so while experimenting make configure set active = true (rather than check the mapping).

@roundsmapp
Copy link

@andrewlewis , I made the changes. It is working great! Thanks again for the help.

@roundsmapp
Copy link

@andrewlewis , good morning! What's the maximum number of outputChannels allowed? 8?

@andrewlewis
Copy link
Collaborator

Yes. See DefaultAudioSink.

@roundsmapp
Copy link

OK, Thank you.

@roundsmapp
Copy link

Hello, @andrewlewis , everything was going fine until I started testing using the 8 channels. I've found a very similar problem as issue #3335 (which is closed, therefore I couldn't place a comment there). For most of the channels, I can clearly control volume and thus have only left or right. But, for some reason, I can't do this for channels 2 and 3 (3rd and 4th).

Is it a bug?

@andrewlewis
Copy link
Collaborator

Could you elaborate a bit on what you're seeing? Specifically: what is the input channel count, what processing are you doing (e.g., multiplying each channel's PCM samples by 0.5), what is the output channel count and how many channels is the platform downmixing to? What is the expected and observed behavior?

It sounds like the platform downmixer is not working as expected.

@roundsmapp
Copy link

The audio file was recorded with 8 channels. So, using ChannelMappingAudioProcessor, I have parameters outputChannels = int[8] and outputChannelVolumes = float[8].
I also have outputChannels[1...8] = 1

By changing the value of outputChannelVolumes, I can control left and right speakers. The intention is to have each pair of channels, 0 and 1; 2 and 3; 4 and 5; 6 and 7 work like "4 channels" in a way that inside each of these pairs the sum of outputChannelVolumes is always 1.0f.

For example, outputChannelVolumes[0] = 0.2f and outputChannelVolumes[1] = 0.8f.

So, when I change outputChannelVolumes[0, 4, 6] from 0.0f to 1.0f, I control the volume of the left speaker. The same is true for outputChannelVolumes[1, 5, 7] to control the volume of the right speaker. All this works perfectly.

But, when I change the value of outputChannelVolumes[2 and 3] nothing happens, i.e., the audio keeps coming from left and right speakers in the same proportion, like if both had been set to 1.0f.

@andrewlewis
Copy link
Collaborator

Do you need to output eight channels, or can you mix down to stereo yourself? The two output channels would each have the sum of contributions from four other channels. I think this was the conclusion on #3335, and seems more reliable than relying on the platform to mix down to stereo in a particular way.

@roundsmapp
Copy link

I have four different audios being played simulteneously and the user needs to be able to control the left/right speaker for each one of those four, while the audio is being played.

So, the only option I could envision so far is to create the audio file with the 8 channels, as I explained above.

Do you have any other suggestion?

@roundsmapp
Copy link

It is working. There is just a small delay between the time I touch the control (on the screen) and the change in the audio response. But this doesn't bother much. It's just that we get used to light speed responses.

Once again, thank you!

@roundsmapp
Copy link

@andrewlewis , back with another problem regarding the use of a customized ChannelMappingAudioProcessor . It started when I tested on Oreo. When playing the loop, it would cut the sound, like if it was processing it in a very slow speed. Even when adjusting the device volume (the own smartphone volume button), this would happen.

So, I decided to use Services/Notification/Foreground. Service is being started, I get the notification and the class that builds the whole ExoPlayer process is also being called.

But then I got a weird error msg:

java.lang.RuntimeException: Unable to start activity ComponentInfo{xxx.PrepareExoPlayer}: 
java.lang.ArrayStoreException: com.google.android.exoplayer2.audio.SilenceSkippingAudioProcessor 
cannot be stored in an array of type xxx.ChannelMappingAudioProcessor[]

The error points to the code line where I have:
simpleExoPlayer = ExoPlayerFactory.newSimpleInstance(defaultRenderersFactory, trackSelector, loadControl);
which is exactly what I had before.

@andrewlewis
Copy link
Collaborator

@roundsmapp For the ArrayStoreException: please make sure you're passing an AudioProcessor[] to the player, not a ChannelMappingAudioProcessor[].

Regarding audio cutting out like processing is too slow: this doesn't sound like the kind of issue that would be fixed by moving the player into a service (though that's the right thing to do anyway if you need to play audio when your app isn't in the foreground).

@roundsmapp
Copy link

The app is designed to be used (play the audio) only in foreground and I coded it to do so. It is working fine in versions prior to Oreo. So, by reading this page https://developer.android.com/about/versions/oreo/background, I understood it would be better to use Services/Foreground on this newer version of Android.

So, if moving to foreground will not supposedly fix the audio cutting, where should I look at? As I said the simple act of changing the device volume affects the audio processing.

@cobarx cobarx mentioned this issue Jun 5, 2018
Merged
@roundsmapp
Copy link

@andrewlewis , do you have any clue about why there is this audio cutting on Oreo, even when doing a volume change through the device buttons?

@andrewlewis
Copy link
Collaborator

By "audio cutting out", do you mean that the audio stops completely, or that it cuts out then resumes?

I don't think it's necessary to use a Service if you app only needs to play audio when it's the foreground activity.

@roundsmapp
Copy link

By "audio cutting" I mean the audio is not being played continuously, it's playing a fraction of a second, stopping a fraction of a second, playing again, stopping again and so on. Not smooth.

I've also noticed that, as far as I have tested, this problem seems to be affecting Samsung Galaxy S series only (since it's on testing phase, I have a limited number of devices I can try).

But I've tested on another brand smartphone running Android 8.0 and it goes well.

Weird.

@roundsmapp
Copy link

@andrewlewis , any clues?

@andrewlewis
Copy link
Collaborator

Perhaps this device is particularly slow and audio processing can't keep up. Do you see audio underruns in logcat when audio cuts out? You could test out a signed release build to see if that makes a difference (it probably won't make much difference on Oreo, though).

@andrewlewis
Copy link
Collaborator

If you attach a bug report taken just after audio was cutting out we can see if there's anything suspicious there.

@roundsmapp
Copy link

I'm not sure I'll be able to produce the bug report so soon, unfortunately. I'm testing the app myself in different smartphones, but the Galaxy S testing is being done by a friend of mine who lives quite far from me. So, in his case, the test is being done without connecting the device to a PC (and he's not an IT guy).

@andrewlewis andrewlewis mentioned this issue Jul 20, 2018
Closed
@roundsmapp
Copy link

roundsmapp commented Jul 26, 2018 via email

@andrewlewis
Copy link
Collaborator

I didn't notice anything suspicious in the logs, or really anything related to media playback. In particular, there's no sign that the audio track is underrunning.

Could you try to reproduce the issue in the ExoPlayer demo app with the minimal changes required to use your audio processor? If you manage to do so please file a new issue and include the steps to reproduce along with a bug report, and we can investigate further. (I think we should move this discussion to a new issue as it seems off-topic for this one.)

@ttm-hoangcm
Copy link

ttm-hoangcm commented Aug 10, 2018 •
edited
Loading

Hello @andrewlewis ,

Sorry to bother you, but can I separate all of a MPEG-DASH stream audio (5.1 surround sound *.mpd file) channels (including R, L, C, SL, SR, LFE channel) and independent control each channel volumes ? What do I need to do that?

Thank you very much.

@andrewlewis
Copy link
Collaborator

@ttm-hoangcm Yes, it should be possible. Please read this thread from the start. It should provide enough information on how to implement your own audio processor that does this. @roundsmapp may also be willing to share their implementation.

This seems like quite a niche feature so we don't currently plan to add it to the core library. If someone can describe an important or popular use case that requires it we can reconsider.

@roundsmapp
Copy link

@andrewlewis , After dozens and dozens of tests, literally, I've notices a weird behaviour, specific for the Galaxy S series. When the audio is started and just after another App is started, i.e., my audio App is played in background and the other App is in foreground, the audio will play smoothly continuously. This was tested during 45 min and there was no problem.

But, as soon as my audio App is returned back to the front, it starts failing. By starts failing I mean the audio is playing like it is in "slow motion", as if the processor is not being able to handle the input/output data in a normal speed. All audio files have 8 channels.

So far I wasn't able to diagnose what is going on.

Any clues?

@andrewlewis
Copy link
Collaborator

@roundsmapp Is your app doing anything else while it's in the foreground? It might be worth removing everything except the player to see if that makes a difference.

If it's still reproducible after doing that, you could try acquiring a wake lock during playback. And perhaps also make sure your app has audio focus. I don't think either of these should make a difference but they are probably worth trying if there aren't any other options.

@roundsmapp
Copy link

I've tried the first scenario, i.e., removing everything except the player. There is no more buttons or any widget or a GUI. Still the problem persists.

I may try audio focus, but it doesn't make sense.

@roundsmapp
Copy link

@andrewlewis , audio focus didn't work. It's really a weird behaviour. Everything works fine if it is in background (not talking about Android Services), but as soon as the App comes to front, it fails.

It looks like it's the way Galaxy S9 with Android 8.0 handles audio processing.

Any more clues?

@roundsmapp
Copy link

@andrewlewis , Wake lock has exactly the same behaviour.

@roundsmapp
Copy link

@andrewlewis , do you have any clues to the problem described above (last three comments)? Thanks

@andrewlewis
Copy link
Collaborator

I can't think of anything I'm afraid. If you haven't tried it already, you could try taking the ExoPlayer demo app and seeing if you can reproduce the issue there with and without your audio processor, playing the same or different content.

@roundsmapp
Copy link

@andrewlewis , the problem still exists with ExoPlayer plain demo app (without using custom audio processor), playing a short audio (less than 20 sec) in loop, ogg format. As said before, the audio file has 8 channels, the phone is a Samsung S9, Android 8.0.

@andrewlewis
Copy link
Collaborator

@roundsmapp It sounds like this isn't related to audio processing. Please could you file a new issue and include the media you're trying to play and a description of the steps to reproduce the issue in the demo app? Thanks.

@roundsmapp
Copy link

@andrewlewis OK, I'll do it. Thank you so far for the support.

@Vitamin-B

This comment has been minimized.

Sign in to view
@google google locked and limited conversation to collaborators Dec 12, 2018
@ojw28 ojw28 closed this as completed Jul 9, 2019
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees

@andrewlewis andrewlewis

Labels
enhancement low priority
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
8 participants
@andrewlewis @ojw28 @mdtuyen @Avetri @platinum7919 @Vitamin-B @roundsmapp @ttm-hoangcm