The renewal maintenance has officially ended for Progress iMacros effective November 30, 2023.
This Wiki site will also no longer be moderated from the Progress side.
Thank you again for your business and support.
Sincerely, The Progress Team
Difference between revisions of "Custom Solutions"
m (Text replacement - "http://www.iopus.com/guides/winscheduler.htm" to "https://wiki.imacros.net/Browser_Automation#Schedule_Tasks") |
|||
(13 intermediate revisions by 3 users not shown) | |||
Line 1: | Line 1: | ||
− | In case you want us at iMacros to develop a solution for you, here's some hints on how to get the project going | + | In case you want us at iMacros to develop a solution for you, here's some hints on how to quickly get the project going. |
== Getting in touch == | == Getting in touch == | ||
− | Getting a free quote is easy: just | + | Getting a free quote is easy: just contact us using the form at http://imacros.net/support/professional-services |
However, the more information we get right from the start, and the more detailed the description is, the quicker we can work on estimating complexity (and thus: costs) of the project. | However, the more information we get right from the start, and the more detailed the description is, the quicker we can work on estimating complexity (and thus: costs) of the project. | ||
Line 12: | Line 12: | ||
What, then, is relevant information? | What, then, is relevant information? | ||
− | We will illustrate the necessary parts with a sample project: | + | We will illustrate the necessary parts with a sample project: fetching data from a real estate property site. |
=== URLs to access === | === URLs to access === | ||
Line 19: | Line 19: | ||
If these are accessible with a login only, '''please provide a (test) login'''. | If these are accessible with a login only, '''please provide a (test) login'''. | ||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
=== Steps to be automated === | === Steps to be automated === | ||
Line 35: | Line 29: | ||
* ... | * ... | ||
− | + | We recommend using a tool like [http://www.techsmith.com/download/snagit/default.asp Snagit] to provide a series of annotated screenshots for visually showing each step in addition to providing a written description. | |
− | |||
− | |||
− | |||
− | |||
=== Input-/Output samples === | === Input-/Output samples === | ||
Line 57: | Line 47: | ||
* custom text files | * custom text files | ||
* ... | * ... | ||
− | |||
− | |||
− | |||
− | |||
=== What programming/scripting language to use === | === What programming/scripting language to use === | ||
Line 73: | Line 59: | ||
In case you want a user interface, please specify what elements of input/output (e.g. free text input, dropdown boxes, information about progress) the interface should show. | In case you want a user interface, please specify what elements of input/output (e.g. free text input, dropdown boxes, information about progress) the interface should show. | ||
− | |||
− | |||
− | |||
− | |||
=== Error handling === | === Error handling === | ||
Line 91: | Line 73: | ||
* What to do with empty result: ignore? Save as "empty"? ... | * What to do with empty result: ignore? Save as "empty"? ... | ||
− | ==== | + | ==Sample Project Description== |
+ | |||
+ | Here's how the necessary information would look like in a sample project (which is described with full source code here: [[VBS_looping]]). | ||
+ | The aim of this project is to automate the extraction of real-estate data from a certain website. | ||
+ | |||
+ | '''Note that iOpus/Ipswitch Inc. is not affiliated with any of the companies that show up in the following sample project description.''' | ||
+ | |||
+ | === URLs to access === | ||
+ | |||
+ | The solution would start at http://realestate.nytimes.com/ | ||
+ | |||
+ | [[Image:Initial_site.png|150px]] | ||
+ | |||
+ | === Steps to be automated === | ||
+ | |||
+ | * On start page: select "New York" and "Manhattan", then click "Go" | ||
+ | *: [[Image:Query1.png|150px]] | ||
+ | * on next page: select "All Manhattan" and click "Search" | ||
+ | *: [[Image:Query2.png|150px]] | ||
+ | * on results page: select "List View" to get the list of data that is to be extracted | ||
+ | *: [[Image:Select_list_view.png|150px]] | ||
+ | |||
+ | * On this list, we need to extract the following data from each of the properties listed: | ||
+ | ** Street | ||
+ | ** District (given below street) | ||
+ | ** Kind of property (same line as district) | ||
+ | ** Price | ||
+ | *: [[Image:What_to_extract.png|150px]] | ||
+ | |||
+ | * And we need to extract all results on the first three pages of result (accessible via the "Next" link") | ||
+ | *: [[Image:Loop_through_pages.png|150px]] | ||
+ | |||
+ | === Input-/Output samples === | ||
+ | |||
+ | There is no input (it is always New York/Manhattan that is worked on). | ||
+ | |||
+ | Extraction file format is CSV and here's a sample of the result file. | ||
+ | |||
+ | "795 FIFTH AVENUE", "Upper East Side", "Co-op", "$70,000,000" | ||
+ | "EAST 68TH STREET", "Upper East Side", "House", "$64,000,000" | ||
+ | |||
+ | |||
+ | === What programming/scripting language to use === | ||
+ | |||
+ | The script is to be run nightly without user interaction. Thus, [https://wiki.imacros.net/Browser_Automation#Schedule_Tasks scheduling] a VBS script is enough for this solution. | ||
+ | |||
+ | === Error handling === | ||
+ | |||
+ | The following features are needed: | ||
− | + | # Report success/failure to log file | |
− | + | # If any of the steps above fail, indicate that the script ended with error. | |
+ | # On error, raise pop-up window |
Revision as of 10:11, 21 November 2018
In case you want us at iMacros to develop a solution for you, here's some hints on how to quickly get the project going.
Getting in touch
Getting a free quote is easy: just contact us using the form at http://imacros.net/support/professional-services
However, the more information we get right from the start, and the more detailed the description is, the quicker we can work on estimating complexity (and thus: costs) of the project.
Project description
What, then, is relevant information?
We will illustrate the necessary parts with a sample project: fetching data from a real estate property site.
URLs to access
As all macros work on web pages, we need to know the URLs of all relevant sites.
If these are accessible with a login only, please provide a (test) login.
Steps to be automated
Keep in mind that we are not as familiar with the site you want to automate, as you are. That's why we need detailed information about
- what links to follow
- to enter what text into what fields
- what checkboxes to check, what radiobuttons to select
- what, exactly, to extract, if extraction is part of the macro (e.g. "extract last name, city and ZIP", and not "get address")
- ...
We recommend using a tool like Snagit to provide a series of annotated screenshots for visually showing each step in addition to providing a written description.
Input-/Output samples
If the project relies on e.g. web queries or on forms to fill, we need a sample of input files that
- tell us about the input layout (i.e. how many columns, what kind of data in what column)
- lets us test the site using this input data. Note that we need valid input data for testing the site!
With respect to the output, we need a sample, too, as it
- gives us more information about what exactly to extract
- tells us how much "formatting" the "raw" extracted data will need
The default format for input/output is CSV files that Excel / OpenOffice.Org et al. can import/export. However, almost every other format is possible, too. Just tell us what you need:
- XML
- SQL datbases
- custom text files
- ...
What programming/scripting language to use
In most cases of automation, VBS scripts are sufficient.
But (e.g. in case you need a "real" user interface) other scripting/programming languages are possible, too, like:
- C#
- Java
- ...
In case you want a user interface, please specify what elements of input/output (e.g. free text input, dropdown boxes, information about progress) the interface should show.
Error handling
When automating websites, several kinds of errors may occur:
- Disconnection
- Server error
- Invalid actions
- "Empty" query results
Please specify how to handle them, e.g.
- Restart script on disconnection
- Check whether actions were performed correctly
- What to do with empty result: ignore? Save as "empty"? ...
Sample Project Description
Here's how the necessary information would look like in a sample project (which is described with full source code here: VBS_looping). The aim of this project is to automate the extraction of real-estate data from a certain website.
Note that iOpus/Ipswitch Inc. is not affiliated with any of the companies that show up in the following sample project description.
URLs to access
The solution would start at http://realestate.nytimes.com/
Steps to be automated
- On start page: select "New York" and "Manhattan", then click "Go"
- on next page: select "All Manhattan" and click "Search"
- on results page: select "List View" to get the list of data that is to be extracted
- On this list, we need to extract the following data from each of the properties listed:
- Street
- District (given below street)
- Kind of property (same line as district)
- Price
- And we need to extract all results on the first three pages of result (accessible via the "Next" link")
Input-/Output samples
There is no input (it is always New York/Manhattan that is worked on).
Extraction file format is CSV and here's a sample of the result file.
"795 FIFTH AVENUE", "Upper East Side", "Co-op", "$70,000,000" "EAST 68TH STREET", "Upper East Side", "House", "$64,000,000"
What programming/scripting language to use
The script is to be run nightly without user interaction. Thus, scheduling a VBS script is enough for this solution.
Error handling
The following features are needed:
- Report success/failure to log file
- If any of the steps above fail, indicate that the script ended with error.
- On error, raise pop-up window