Compressing a blob

Is there something that will compress a ‘blob’ before store it in a database?

Have a user of my application that want to use my application is a way not attended. Not that it does not work.

Have a window/feature where the user can store Notes. The primary field is a FTC (BKeeney’s ) control which is stored in a database table, stored as a blob. The user has figured out that he can put images in that field. This work ok, for awhile, then when he tries to add another image it crashes. Have not figured out why but I think it may have something to do with memory.

I’m not asking about the crash. But about the ‘blob’ and the size of the table. As you know images files these days are getting quite large, and adding a few images and and your database get bigger and bigger. That may or may not be a problem, but it can’t be good. My solution is to save the path to the file but is not what the user wants, but may have to live with.

Is there a way to compress the blob before storing, and of course uncompressing when reading from the database?

Not knowing what causes the crash is important

Compressing it may / may not solve the problem - or may just mask the problem until they add yet another large image

That true. It’s just not what my search it for with this postX!

To compress it you could certainly zip the data and place it in the blob field
MBS probably has something to do this
zip on the way in & unzip on the away out

But the “add another image and it crashes” suggests that the database is NOT the problem

Agree.
The first problem is getting where the crash happen. The drag, drop, save, new or just what from the user. Other than “It don’t work”!

[quote=122163:@Jim Smith]Is there something that will compress a ‘blob’ before store it in a database?
Is there a way to compress the blob before storing, and of course uncompressing when reading from the database?[/quote]

being an image, do you want lossy or lossless?

for lossless, use OS X’s built-in zlib:

soft declare function zlibcompress lib zlibPath alias “compress” (dest as Ptr, ByRef destLen as Uint32, source as Ptr, sourceLen as UInt32) as Integer

dim output as new MemoryBlock(12 + 1.002*inputSizeInBytes)
dim outputSize as UInt32 = output.Size

pLastErrorCode = zlibcompress(output, outputSize, input, inputSizeInBytes)
if pLastErrorCode = 0 then
compressedSize = outputSize
return output
else
return nil
end if

soft declare function zlibuncompress lib zlibPath alias “uncompress” (dest as Ptr, ByRef destLen as UInt32, source as Ptr, sourceLen as Uint32) as Integer
dim destLength as UInt32 = outputBufferSizeInBytes
dim sourcePtr as Ptr = input
dim destPtr as Ptr = outputBuffer

pLastErrorCode = zlibuncompress(destPtr, destLength, sourcePtr, compressedSize)

return pLastErrorCode

For lossy, I use Christian’s GMImageMBS to save as JPEG2000 (slightly better compression with slightly better quality compared to JPEG)

P.

[quote=122163:@Jim Smith]Is there a way to compress the blob before storing, and of course uncompressing when reading from the database?
[/quote]

You can shell to Mac OS X zip command. It is fairly straightforward. I use it in one my apps, and it is quite reliable.

https://developer.apple.com/library/mac/documentation/Darwin/Reference/ManPages/man1/zip.1.html

Thanks guys.
This is for Mac and Windows, so I think I will use MBS as I already have the plugin.

I’d recommend saving it in the FTC’s XML format. It’s text and it’s in the native format for FTC. If you need it searchable perhaps you save the XML version and the text version so it can be searchable via query.

If you’re using SQLite you won’t need to worry about length. Obviously other databases will have different field requirements.

I would agree. Database code is generally rock solid. You need to get more information from the user. If Mac OS X perhaps you can get a stack trace to help figure it out. If in Windows you’ll need to do more legwork.

I use TTZipPackage for easy cross platform zip/unzip.

You need to get a license if you’re going to snoop around the zip one file at a time (which is really cool, but I can’t afford to do that yet) but to just compress/decompress to a folder it’s free.

Hi everybody!

I’m looking for a method to compress blob columns in SQLite, just as asked in this topic. But I want to do so for Windows.

I looked at the plugins offered, but these are too complex and difficult to understand, since I’m quite new to all this. However I understand the way the declare is working for OS X (at least it makes sense to me).

So I wonder if there is a similar method like the declare way for OS X also for Windows? Can someone help me out there? Looking forward any help you’d can provide. :slight_smile:

Greetings, Frank.

[quote=166351:@Frank Foerster]Hi everybody!

I’m looking for a method to compress blob columns in SQLite, just as asked in this topic. But I want to do so for Windows.

I looked at the plugins offered, but these are too complex and difficult to understand, since I’m quite new to all this. However I understand the way the declare is working for OS X (at least it makes sense to me).

So I wonder if there is a similar method like the declare way for OS X also for Windows? Can someone help me out there? Looking forward any help you’d can provide. :)[/quote]

You could shell to COMPRESS

[code]COMPRESS [-R] [-D] [-S] [ -Z | -ZX ] Source Destination
COMPRESS -R [-D] [-S] [ -Z | -ZX ] Source [Destination]

Description:
Compresses one or more files.

Parameter List:
-R Rename compressed files.

-D Update compressed files only if out of date.

-S Suppress copyright information.

-ZX LZX compression. This is default compression.

-Z MS-ZIP compression.

Source Source file specification. Wildcards may be
used.

Destination Destination file | path specification.
Destination may be a directory. If Source is
multiple files and -r is not specified,
Destination must be a directory.

Examples:
COMPRESS temp.txt compressed.txt
COMPRESS -R .
COMPRESS -R *.exe *.dll compressed_dir[/code]

Use the -Z option, then to unpack use the command EXPAND.

I do not know if COMPRESS is included in latest Windows versions (in my Win7 Pro it is not).

An alternative (before shelling sth out) could be in using Declare with COM in WIndows.
Each Windowversion since WIndows XP has an built in Zipper. I’ve found these lines for VBS but it should be easy to connect to COM object and modify it for xojo.

Using SourceFolder as Source and ZipFile as Target:


CreateObject("Scripting.FileSystemObject").CreateTextFile(ZipFile, True).Write "PK" & Chr(5) & Chr(6) & String(18, vbNullChar)

Set objShell = CreateObject("Shell.Application")
Set source = objShell.NameSpace(SourceFolder).Items

objShell.NameSpace(ZipFile).CopyHere(source)

[quote=166386:@Tomas Jakobs]I do not know if COMPRESS is included in latest Windows versions (in my Win7 Pro it is not).
[/quote]

You are right, it is no longer included, and not present in Windows 8.1 or Windows 10.

The alternative would be to bundle a helper. I tried to access sourceforge.net but it is down at this moment.

7-Zip is in LPGL and could possibly be adequate :
http://www.7-zip.org/

Sorry – Had to be done –

On a system with NTFS, the native utility is COMPACT. You can compress a single file or a whole folder. You can expand the file with the command EXPAND link

edit: while the link is for Windows XP, COMPACT is verified to work on Windows 8.1.

Have you tried pushing it with your finger?
Sorry - that also had to be said :slight_smile:

Also needs to be said: Using a plugin will be much simpler and cross-platform. Spend just a little time to learn how to use plugins. It will open up a whole world of functionality for you.

You are so right, Tim.

This is probably the simplest way :
https://www.monkeybreadsoftware.net/pluginpart-zip.shtml

I don’t like using plugins since I want to have the control about what I write and want to understand how it works. Plugins appear like black boxes to me that can easily break.

Tomas Jakobs , can you please closer explain your source code ? I don’t know how to convert this into Xojo.

[code]CreateObject(“Scripting.FileSystemObject”).CreateTextFile(ZipFile, True).Write “PK” & Chr(5) & Chr(6) & String(18, vbNullChar)

Set objShell = CreateObject(“Shell.Application”)
Set source = objShell.NameSpace(SourceFolder).Items

objShell.NameSpace(ZipFile).CopyHere(source)[/code]