Diablo 3 doesn’t allow the player to search for an item by clicking on it while in auction house and list it like the World of Warcraft auction house does. This is probably for making the auction house more interesting and easy for players to hunt cheap items.
The items are listed after a search by those with the highest stats, for a weapon, the auction house will list the ones with the highest DPS if there are no other search criteria like vitality or strength.
You can set a price limit before hitting the search button from “Max buyout” field.
How to determine the price for legendary and set items in Diablo 3? There is another field near “Max buyout”, “Legendary or set item”, you can write the name or the first letters for a legendary or set item (has auto-complete) and view the list that will display those legendary or set item. You can browse to the last page to check the cheapest price for that item. You can make an idea for how much you should sell your items.
For rare items, the only way to determine the price is to play the auction house, have a good knowledge of the game, of the items and stats, of how are working and know what exactly are the players looking for, the best stats for their class / combo / skills. For the barbarian class for example, nobody will sell many 2 handed weapons as these are not viable or popular for end game.