first I would make sure that you dont keep the flag for gathering debug information turned on in the release as this just hangs around in memory as it builds up unless you read it out.
For an RTSP stream I can get it connect but after a few short minutes of streaming I get some stream error written to stderr but not an error event that I have figured out how to trap yet and then the stream halts and goes to 100% of a CPU and just sits there.
Even when it is working it seems to leak memory and after a while you have to close up and nil all the structures and restart with fresh ones or youll end up with all the memory. I havent tested this yet in that project, but it was true in the past.
Its quite a bit of work to fix the loading paths for all the plugins. the otool -L output for the previous version that worked for me, or at least loaded properly looked like this:
otool -L ./libvlc.5.dylib
./libvlc.5.dylib:
@loader_path/lib/libvlc.5.dylib (compatibility version 11.0.0, current version 11.1.0)
@loader_path/…/lib/libvlccore.8.dylib (compatibility version 9.0.0, current version 9.1.0)
/usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 159.1.0)
/usr/lib/libiconv.2.dylib (compatibility version 7.0.0, current version 7.0.0)
otool -L ./libvlccore.8.dylib
./libvlccore.8.dylib:
@loader_path/lib/libvlccore.8.dylib (compatibility version 9.0.0, current version 9.1.0)
/usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 159.1.0)
/usr/lib/libiconv.2.dylib (compatibility version 7.0.0, current version 7.0.0)
/System/Library/Frameworks/CoreFoundation.framework/Versions/A/CoreFoundation (compatibility version 150.0.0, current version 635.21.0)
/System/Library/Frameworks/CoreServices.framework/Versions/A/CoreServices (compatibility version 1.0.0, current version 53.0.0)
but the new versions which wont load are built differently and they look like this now:
otool -L ./libvlc.5.dylib
./libvlc.5.dylib:
@rpath/libvlc.dylib (compatibility version 12.0.0, current version 12.0.0)
@rpath/libvlccore.dylib (compatibility version 10.0.0, current version 10.0.0)
/usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 1252.50.4)
/usr/lib/libiconv.2.dylib (compatibility version 7.0.0, current version 7.0.0)
./libvlccore.9.dylib
./libvlccore.9.dylib:
@rpath/libvlccore.dylib (compatibility version 10.0.0, current version 10.0.0)
/usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 1252.50.4)
/usr/lib/libiconv.2.dylib (compatibility version 7.0.0, current version 7.0.0)
/System/Library/Frameworks/CoreFoundation.framework/Versions/A/CoreFoundation (compatibility version 150.0.0, current version 1452.23.0)
/System/Library/Frameworks/CoreServices.framework/Versions/A/CoreServices (compatibility version 1.0.0, current version 822.31.0)
that @rpath does not work outside of the regular VLC app somehow. Or perhaps there is a way to change the links as we load them into XTension, but I dont think so. The libraries need to be able to find each other. Or perhaps we can actually set that @rpath in the environment for our app and have it come out right? I used the install_name_tool to change the names and links of the libraries so that they could find each other.
I can share exactly what I did, at least it will let the libraries load and then you can figure out if you can use it for what you want to do. Updating libvlc.5.dylib requires changing both its own id parm and one of the links like this: I ran these from inside the same folder as the libraries:
install_name_tool -id @executable_path/lib/libvlc.dylib ./libvlc.5.dylib
install_name_tool -change @rpath/libvlccore.dylib @executable_path/lib/libvlccore.dylib ./libvlc.5.dylib
after which the otool -L output should look like:
otool -L ./libvlc.5.dylib
./libvlc.5.dylib:
@executable_path/lib/libvlc.dylib (compatibility version 12.0.0, current version 12.0.0)
@executable_path/lib/libvlccore.dylib (compatibility version 10.0.0, current version 10.0.0)
/usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 1252.50.4)
/usr/lib/libiconv.2.dylib (compatibility version 7.0.0, current version 7.0.0)
and the libvlccore library only needs its id changed like this:
install_name_tool -id @executable_path/lib/libvlccore.dylib ./libvlccore.9.dylib
unfortunately then you have to also update ALL the plugins that you want to use too. The commands are like:
install_name_tool -change @rpath/libvlccore.dylib @executable_path/lib/libvlccore.dylib ./liba52_plugin.dylib
install_name_tool -change @rpath/libvlccore.dylib @executable_path/lib/libvlccore.dylib ./libequalizer_plugin.dylib
install_name_tool -change @rpath/libvlccore.dylib @executable_path/lib/libvlccore.dylib ./libmpgv_plugin.dylib
etc…
and so forth, i just built myself a batch file to run them all as there are a LOT of them. Probably I dont need them all and could remove most of them but I havent experimented with that.
if you do that and just hard copy them into the app in the same format as the MBS example creates the folders and sym links then they will load. They still wont work yet though as youll get an error about an incompatible plugin cache as the version that the app loads and caches is obviously not the same, so you need to tell VLC at startup not to use the plugin cache. You can do that in the init methods where you startup the libraries and pass some command line parms to the constructor of the VLCInstance. The parms Im using are basically the same as the MBS example with the addition of –no-plugins-cache like this:
Dim vlcargs() As String = Array( app.ExecutableFile.NativePath, _
“–no-video-title-show”,_
“–verbose=0”,_
“–no-media-library”,_
“–no-sub-autodetect”,_
“–ignore-config”,_
"–no-plugins-cache)
VLCInstance = new VLCInstance( vlcargs)
NOW it should load and actually run the examples. You cant use the MBS example code to debug copy libraries or whatever it was called as it will just copy the unaltered libraries from inside the VLC app and these altered ones wont run from inside the VLC app, or actually maybe they would I didnt try that… so I rebuilt that example code to copy them from a different location relative to my project folder and then create the same symlinks as is done in the MBS example.
I THINK that was all I had to do to get them to load the latest versions and actually run 
If it doesnt work I can go through it all again but thats my notes. After you get it actually to load and run then you can figure out if its going to work OK for your application or if its going to crash out or leak too much memory or whatever else.
The most frustrating part of it for me is that after all that even when its properly playing the RTSP stream there is about a 2 second delay from reality. None of the parms to set the stream to realtime or reduce the buffer size seem to make any difference, which makes it not so great for my purposes anyway. I can probably find ways around the memory leak and the stream crashing and going to 100%CPU by just restarting everything. Which will mean that there will be 2 or 3 second gaps in any recordings Im making as it restarts the stream but still it might work. The 2 second delay Im not sure I can do anything about. Im not aware of any other way to handle an RTSP stream on the Mac right now. I may explore handling it natively myself if I have to, but ugh… Quick Time used to handle it just fine, but you cant link those libraries in anymore. I have to use xojo 2013 in order to link in the MBS quick time plugin and that does work, but thats so old I cant consider that a real solution going forward. AVFoundation does not do RTSP only HTTP streaming. Its been a while since Ive looked into that though, so maybe something else has changed Sometimes a project is just a honey pot that never lets you get it done and reading RTSP streams has been one for me definitely.