Wed 6 Feb 2008
Now we’re ready to create an Extract to link our Source to our Target. From the main menu go ahead and click Create from the Extract menu.
Creating an Extract
- Step one asks for the name, Source and Target. It would be a good idea to name it the same as your Source to keep up with it. Choose the cooresponding source and our target we created in the previous tutorial. Click Apply.
- The next step is for Batch Parameters. I kept the defaults except Batch_Size, set it to 100. This setting is really up to you. This is 10 by default but runs fine for me at 100. Clicked Apply.
- Step three in my case warns me my provider does not have a media server, which is fine since we will be downloading everything anyway. Click Apply.
- Step four ask about refreshing values. Set this to false. Click Apply.
- Step five is very important. Here is where you tell it which fields go to which database table columns. I generally place my mls number, class, type, status in the first few fields. Also it is a very good idea to write these down! As you create more Sources and Extracts you will want to keep the columns as similar as possible to search across all types of listings. (keep price in the same column so no matter if you’re searching Land or Condos, its always going to be in that column).
- Step six is the same thing but with the image table. You should be able to include all the fields, but keep these the same throughout all of your extracts.
Now we have our Extract. Our next step is to alter our batch file to retrieve listings with an Active Status only, and execute our batch via command line.
- RETS Retrieval with PHP and VieleRETS: Part 1, Introduction
- RETS Retrieval with PHP and VieleRETS: Part 2, Creating a Source
- RETS Retrieval with PHP and VieleRETS: Part 3, Creating a Target
- RETS Retrieval with PHP and VieleRETS: Part 4, Creating an Extract
- RETS Retrieval with PHP and VieleRETS: Part 5, Executing the Batch
- RETS Retrieval with PHP and VieleRETS: Part 6, Creating a Cron Job

February 7th, 2008 at 10:39 pm
[...] RETS Retrieval with PHP and VieleRETS: Part 2, Creating a Source [...]
February 7th, 2008 at 10:39 pm
[...] RETS Retrieval with PHP and VieleRETS: Part 4, Create an Extract [...]
February 18th, 2008 at 7:46 pm
Thanks for the great tutorial Trent. Question: Why not just name the columns in your database something more meaningful than c1,c2,c3? Seems like that just adds a layer of confusion. Wouldn’t it just be easier to set the table up with names that make sense and just take a little more time in mapping them correctly?
February 18th, 2008 at 8:50 pm
I agree with you completely. I’m not sure if it will allow it though. When setting it up it doesn’t really give you an option of what the columns are, but it simply refers to them by c1,c2,c3. So by that I don’t know if it will break any sql within vieleRETS. It may do some sort of scaffolding to read the columns names? I would love to know if it breaks myself.
February 18th, 2008 at 11:27 pm
I just tried it with different names than c1,c2,c3 and it seemed to work fine. It will only allow the columns that you have created in your database, but I don’t see in the documentation where it says you need to name your columns like that. Is there a link that recommends this naming convention? I’m new to this so I’m just wading through it.
February 19th, 2008 at 7:55 am
Excellent, I actually might go back and change mine and change the blog entry as well. Will it allow more than 24 columns? I couldn’t really find much documentation on it and vieleRETS was suggesting c1,c2,c3..c24 so I took it as thats what it wanted.
The entire time I struggled throuh this and once I got it to work for the most part I decided I would atleast document that much in hopes of helping someone else. I will try and keep this tutorial up to date if any more improvements come along.
February 19th, 2008 at 10:34 pm
Yes, much appreciated. I’m amazed at how little documentation there is out there. If you check the VeileRets Mailing list for February you’ll see that I’ve been hammering them with questions the last few days so I’m definitely getting a better handle on it. I think part of the problem is that they have opted for a mailing list rather than a forum so everyone is in the dark.
Anyway, yes you can have as many columns as you want. Most RETS servers will have hundreds of columns. You would have gotten the c1,c2 naming convention and 24 fields from the auto-create documentation. Viele allows you to automatically create tables with a specified amount of fields and it will name them as such. So that would remove the need to manually create your table of c1,c2,c3. If you read the mailing list you’ll see that I’ve proposed auto-creation using the DBName or something more friendly, yet still automatic and structured.
February 19th, 2008 at 11:04 pm
A forum would be much better. Yeah the entire table issue was very ambiguous. Thanks for the follow up.
I actually just noticed a problem with my feed. It seems to be only pulling 2500 listings. I have LIMIT set to NONE and I’ve tried turning pagination on and off but no luck. Also only one image per listing is pulling as well. I don’t know if that is the same problem or not. Did you have this trouble?
February 20th, 2008 at 12:09 am
FYI…I answered this on the Viele mailing list.
February 20th, 2008 at 12:10 am
There must be a delay before the mailing list updates. Here is what I wrote…
Hello Trent:
That is likely a limitation imposed by your RETS server. I was initially limited to 500 records, but given privileges for unlimited downloads after speaking with the MLS RETS contact.
As for the image issue, uncheck “Media Multipart” and see if that fixes the problem.
February 20th, 2008 at 8:10 am
Thanks David for your response. I will run the update again tonight with the multipart option and keep my fingers crossed. As far as the other part I that is they case they just have a limit on my account.
March 12th, 2008 at 7:26 am
As an update to the issue with being cut off at 2500 listings it was in fact by the provider. the Offset Adjustment was what needed to be changed. Each batch had to run in increments of 2500, so the first batch offset adjustment was 0, then 2500, 5000, 7500, etc until its updated all the listings
April 2nd, 2008 at 10:56 am
Thanks for the excellent tutorial. It helped alot.
I’m not sure if you ran into this, but I’m trying to get an offset in vieleRETS, and I’ve entered 18910 into the batch control file, and viele just says “offset option ignored.” Do you know how to set the offset option in vieleRETS?
Thanks
April 2nd, 2008 at 11:51 am
I did have to use offset, but it worked as expected. Are you sure your offset isn’t greater than the number of items? I could see that possibly being ignored.. but not sure. As in the comment above my provider was limiting me at 2500 items per request, so I had to execute the batch with different offset in steps of 2500 starting from 0: 0,2500,5000,7500.. Also pagination may need to be turned on?